Early warning and collision avoidance

ABSTRACT

Among other things, equipment is located at an intersection of a transportation network. The equipment includes an input to receive data from a sensor oriented to monitor ground transportation entities at or near the intersection. A wireless communication device sends to a device of one of the ground transportation entities, a warning about a dangerous situation at or near the intersection, there is a processor and a storage for instructions executable by the processor to perform actions including the following. A machine learning model is stored that can predict behavior of ground transportation entities at or near the intersection at a current time. The machine learning model is based on training data about previous motion and related behavior of ground transportation entities at or near the intersection. Current motion data received from the sensor about ground transportation entities at or near the intersection is applied to the machine learning model to predict imminent behaviors of the ground transportation entities. An imminent dangerous situation for one or more of the ground transportation entities at or near the intersection is inferred from the predicted imminent behaviors. The wireless communication device sends the warning about the dangerous situation to the device of one of the ground transportation entities.

This application is entitled to the benefit of the filing date of U.S.provisional patent application 62/644,725, filed Mar. 19, 2018, theentire contents of which are incorporated here by reference.

BACKGROUND

This description relates to early warning and collision avoidance.

Collision avoidance systems have become abundant. King et al. (US patentpublication 2007/0276600 A1, 2007), for example, described placingsensors ahead of an intersection and applying a physics-based decisionrule to predict if two vehicles are about to crash at the intersectionbased on heading and speed.

In Aoude et al. (U.S. Pat. No. 9,129,519 B2, 2015, the entire contentsof which are incorporated here by reference) the behavior of drivers ismonitored and modeled to allow for the prediction and prevention of aviolation in traffic situations at intersections.

Collision avoidance is the main defense against injury and loss of lifeand property in ground transportation. Providing early warning ofdangerous situations aids collision avoidance.

SUMMARY

In general, in an aspect, equipment is located at an intersection of atransportation network. The equipment includes an input to receive datafrom a sensor oriented to monitor ground transportation entities at ornear the intersection. A wireless communication device sends to a deviceof one of the ground transportation entities, a warning about adangerous situation at or near the intersection, there is a processorand a storage for instructions executable by the processor to performactions including the following. A machine learning model is stored thatcan predict behavior of ground transportation entities at or near theintersection at a current time. The machine learning model is based ontraining data about previous motion and related behavior of groundtransportation entities at or near the intersection. Current motion datareceived from the sensor about ground transportation entities at or nearthe intersection is applied to the machine learning model to predictimminent behaviors of the ground transportation entities. An imminentdangerous situation for one or more of the ground transportationentities at or near the intersection is inferred from the predictedimminent behaviors. The wireless communication device sends the warningabout the dangerous situation to the device of one of the groundtransportation entities.

Implementations may include one or a combination of two or more of thefollowing features. The wireless communication device sends the warningabout the dangerous situation to a sign or other infrastructurepresentation device. The warning includes an instruction or commandcapable of controlling a specific ground transportation entity. Theequipment includes a roadside equipment. There is a housing for theequipment and the sensor is attached to the housing. The warning is sentby broadcasting the warning for receipt by any of the groundtransportation entities at or near the intersection. The machinelearning model includes an artificial intelligence model. The trainingdata and the motion data include at least one of speed, location, orheading. The training data and motion data may also include intent,posture, direction of look, or interaction with other vulnerable roadusers, such as in a group. The processor is configured to be capable ofgenerating the machine learning model at the equipment. The trainingdata is stored at the equipment. The intersection includes anon-signalized intersection. The intersection includes a signalizedintersection. The transportation network includes a road network. Theground transportation entities include vulnerable road users. The groundtransportation entities include vehicles. The imminent dangeroussituation includes a collision or a near miss. The ground transportationentities include pedestrians crossing a road at a crosswalk. There isanother communication device to communicate with a central server. Thedevice of one of the ground transportation entity includes a mobilecommunication device.

In general, in an aspect, equipment is located at an intersection of thetransportation network. The equipment includes an input to receive datafrom a sensor oriented to monitor ground transportation entities at ornear the intersection. A wireless communication device sends to a deviceof one of the ground transportation entities, a warning about adangerous situation at or near the intersection. There is a processorand a storage for instructions executable by the processor to store amachine learning model that can predict behavior of groundtransportation entities at or near the intersection at a current time.The machine learning model is based on training data about previousmotion and related behavior of ground transportation entities at or nearthe intersection. Current motion data received from the sensor aboutground transportation entities at or near the intersection is applied tothe machine learning model to predict imminent behaviors of the groundtransportation entities, including a ground transportation entity thedevice of which cannot receive a warning from the wireless communicationdevice. An imminent dangerous situation for a ground transportationentity the device of which can receive a warning from the wirelesscommunication device is inferred. The imminent dangerous situation isthe result of predicted imminent behavior of the ground transportationentity that cannot receive the warning. The warning about the dangeroussituation is sent to the device of the ground transportation entity thatcan receive the warning from the wireless communication device.

Implementations may include one or a combination of two or more of thefollowing features. The equipment includes a roadside equipment. Thereis a housing for the equipment and the sensor is attached to thehousing. The warning is sent by broadcasting the warning for receipt byany of the ground transportation entities at or near the intersectionthat can receive the warning. The machine learning model includes anartificial intelligence model. The intersection includes anon-signalized intersection. The intersection includes a signalizedintersection. The transportation network includes a road network. Theground transportation entities include vulnerable road users. The groundtransportation entities include vehicles. The imminent dangeroussituation includes a collision. The ground transportation entity thedevice of which cannot receive the warning from the wirelesscommunication device includes a vehicle. The ground transportationentity the device of which can receive the warning from the wirelesscommunication device includes a pedestrian crossing a road at acrosswalk. There is another communication device to communicate with acentral server. The device of one of the ground transportation entityincludes a mobile communication device.

In general, in an aspect, on board a road vehicle traveling in a groundtransportation network messages and data are received including messagesfrom external sources about location, motion, and state of other groundtransportation entities, data from on board sensors about road anddriving conditions and about the locations of static objects and movingground transportation entities in the vehicle surroundings, data aboutquality of driving by a driver of the road vehicle, and basic safetymessages from other ground transportation entities and personal safetymessages from vulnerable road users. The received data and messages arefused and applied to an artificial intelligence model to predict anaction of a driver of the road vehicle or of a vulnerable road user or acollision risk for the road vehicle or both.

Implementations may include one or a combination of two or more of thefollowing features. The road vehicle creates a map of the static objectsand moving ground transportation entities in the vicinity of the roadvehicle. The driver of the road vehicle is alerted to a collision risk.The collision risk is determined based on probabilities of predictedtrajectories of nearby other moving ground transportation entities.Basic safety messages and personal safety messages are filtered toreduce the number of alerts provided to the driver of the road vehicle.

In general, in an aspect, electronic sensors located in a vicinity of acrosswalk that crosses a road are used to monitor an area in or nearbythe crosswalk. The electronic sensors generate motion data aboutvulnerable roadway users who are in or nearby the pedestrian crosswalk.The generated motion data is applied to a machine learning model runningin equipment located in the vicinity of the crosswalk to predict thatone of the vulnerable roadway users is about to enter the crosswalk.Before the vulnerable roadway users enters the crosswalk, a warning iswirelessly transmitted to at least one of: a device associated with thevulnerable roadway user, or a device associated with another groundtransportation entity that is approaching the crosswalk on the road.

Implementations may include one or a combination of two or more of thefollowing features. The equipment includes a roadside equipment. Thevulnerable roadway user includes a pedestrian, animal, or cyclist. Thedevice associated with the vulnerable roadway user includes a smartwatch or other wearable, a smart phone, or another mobile device. Theother ground transportation entity includes a motorized vehicle. Thedevice associated with the other ground transportation entity includes asmart phone or another mobile device. The machine learning model isprovided to the equipment located in the vicinity of the crosswalk by aremote server through the Internet. The machine learning model isgenerated at the equipment located in the vicinity of the crosswalk. Themachine learning model is trained using motion data generated by thesensors located in the vicinity of the crosswalk. Motion data generatedby the sensors located in the vicinity of the crosswalk is sent to aserver for use in training the machine learning model. The motion datagenerated by the sensors located in the vicinity of the crosswalk issegmented based on corresponding zones in the vicinity of the crosswalk.The electronic sensors are used to generate motion related datarepresenting physical properties of the vulnerable road user. Trajectoryinformation about the vulnerable road user is derived from motion datagenerated by the sensor.

In general, in an aspect, electronic sensors located in a vicinity of anintersection of a ground transportation network are used to monitor theintersection and approaches to the intersection. The electronic sensorsgenerate motion data about ground transportation entities moving on theapproaches or in the intersections. One or more of the groundtransportation entities are not capable of sending basic safety messagesto other ground transportation entities in the vicinity of theintersection. Based on the motion data generated by the electronicsensors, virtual basic safety messages are sent to one of more of theground transportation entities that are capable of receiving themessages. The virtual basic safety messages incorporate informationabout one or more of the ground transportation entities that are notcapable of sending basic safety messages. The incorporated informationin each of the virtual basic safety messages includes at least one ofthe location, heading, speed, and predicted future trajectory of one ofthe ground transportation entities that are not capable of sending basicsafety messages.

Implementations may include one or a combination of two or more of thefollowing features. The equipment includes a roadside equipment. Theincorporated information includes a subset of information that would beincorporated in a basic safety message generated by the groundtransportation entity if it were capable of sending basic safetymessages. The generated motion data is applied to a machine learningmodel running in equipment located in the vicinity of the intersectionto predict a trajectory of the ground transportation entity that is notcapable of sending basic safety messages. One of the groundtransportation entities includes a motorized vehicle. The machinelearning model is provided to the equipment located in the vicinity ofthe intersection by a remote server through the Internet. The machinelearning model is generated at the equipment located in the vicinity ofthe intersection. The machine learning model is trained using motiondata generated by the sensors located in the vicinity of theintersection. Motion data generated by the sensors located in thevicinity of the intersection is sent to a server for use in training themachine learning model.

In general, in an aspect, electronic sensors located in a vicinity of anintersection of a ground transportation network are used to monitor theintersection and approaches to the intersection. The electronic sensorsgenerate motion data about ground transportation entities moving on theapproaches or in the intersections. Distinct virtual zones are definedin the intersection and the approaches to the intersection. The motiondata is segmented according to corresponding virtual zones to which thegenerated motion data relates. The generated motion data is applied foreach of the respective segments to a machine learning model running inequipment located in the vicinity of the intersection to predict animminent dangerous situation in the intersection or one of theapproaches involving one or more of the ground transportation entities.Before the imminent dangerous situation becomes an actual dangeroussituation, a warning is wirelessly transmitted to a device associatedwith at least one of the involved ground transportation entities.

Implementations may include one or a combination of two or more of thefollowing features. The equipment includes a roadside equipment. Thedevice associated with each of the ground transportation entitiesincludes a wearable, a smart phone, or another mobile device. One of theground transportation entities includes a motorized vehicle. The machinelearning model is provided to the equipment located in the vicinity ofthe intersection by a remote server through the Internet. The machinelearning model is generated at the equipment located in the vicinity ofthe intersection. The machine learning model is trained using motiondata generated by the sensors located in the vicinity of theintersection. The motion data generated by the sensors located in thevicinity of the intersection is sent to a server for use in training themachine learning model. The electronic sensors are used to monitor anarea in or nearby a crosswalk that crosses one of the approaches to theintersection. The electronic sensors are used to generate motion relateddata representing physical properties of a vulnerable road user in thevicinity of the crosswalk. Trajectory information about the vulnerableroad user is derived from motion data generated by the sensor. There isa machine learning model for each of the approaches to the intersection.A determination is made whether to transmit the warning based also onmotion data generated by sensors with respect to another nearbyintersection. A determination is made whether to transmit the warningbased also on information received from ground transportation entitiesmoving on the approaches or in the intersection. The intersection issignalized and information about the state of the signals is received.The intersection is not signalized and is controlled by one or moresigns. The defined virtual zones include one or more approachescontrolled by the signs. The signs include a stop sign or a yield sign.One of the ground transportation entities includes a rail vehicle.

In general, in an aspect, equipment is located in or on a groundtransportation entity. The equipment includes an input to receive datafrom sensors in or on the ground transportation entity and oriented tomonitor nearby features of a ground transportation network and otherinformation about a context in which the ground transportation entity istraversing the ground transportation network. A wireless communicationdevice receives information about the context. A signal processorapplies signal processing to data from the sensor and other informationabout the context. There is a processor and a storage for instructionsexecutable by the processor to perform actions that include thefollowing: store a machine learning model that can predict behavior ofan operator of the ground transportation entity and intent and movementof other ground transportation entities in the vicinity, and apply thecurrent received data from the sensors and other information about thecontext to predict behavior of the operator and the intent and movementof other ground transportation entities in the vicinity.

Implementations may include one or a combination of two or more of thefollowing features. The equipment includes a roadside equipment. Theinstructions are executable by the processor to monitor users oroccupants of the ground transportation entity. The other informationabout the context includes emergency broadcasts, traffic and safetymessages road side equipment, and messages about safety, locations, andother motion information from other ground transportation entities. Thesensors include cameras, range sensors, vibration sensors, microphones,seating sensors, hydrocarbon sensors, sensors of volatile organiccompounds and other toxic materials, and kinematic sensors orcombinations of them. The instructions are executable by the processorto filter received alerts that the vehicle receives by applying thealerts to a machine learning model to predict which alerts are importantin the current location, environmental conditions, driver behavior,vehicle health and status, and kinematics.

In general, in an aspect, motion data are acquired for unconnectedground transportation entities moving in a transportation network.Virtual safety messages incorporating information about the motion datafor the unconnected ground transportation entities at sent to connectedground transportation entities in the vicinity of the unconnected groundtransportation entities.

Implementations may include one or a combination of two or more of thefollowing features. the virtual safety messages are substitutes forsafety messages that would be sent by the unconnected groundtransportation entities if they were connected. The unconnected groundtransportation entities include vehicles and the virtual safety messagesare substitutes for basic safety messages. The unconnected groundtransportation entities include vulnerable road users and the virtualsafety messages are substitutes for personal safety messages. The motiondata are detected by infrastructure sensors.

In general, in an aspect, equipment located at an intersection of atransportation network includes inputs to receive data from sensorsoriented to monitor ground transportation entities at or near theintersection. The data from each of the sensors represents at least onelocation or motion parameter of at least one of the groundtransportation entities. The data from each of the sensors is expressedin a native format. The data received from at least two of the sensorsis inconsistent with respect to the location or motion parameters or thenative formats or both. There is a storage for instructions executableby a processor to convert the data from each of the sensors into datahaving a common format independent of the native formats of the data ofthe sensors. The data having the common format is incorporated into aglobal unified representation of the ground transportation entitiesbeing monitored at or near the intersection. The global unifiedrepresentation includes the location, speed, and heading of each of theground transportation entities. Relationships of locations and motionsof two of the ground transportation entities are determined using theglobal unified representation. A dangerous situation is predictedinvolving the two ground transportation entities, and a message is sentto at least one of the two ground transportation entities alerting it tothe dangerous situation.

Implementations may include one or a combination of two or more of thefollowing features. The sensors include at least two of: radar, lidar,and a camera. The data received from one of the sensors includes imagedata of a field of view at successive moments. The data received fromone of the sensors includes points of reflection in 3D space. The datareceived from one of the sensors includes distance from the sensor andspeed. The global unified representation represents locations of theground transportation entities in a common reference frame. Two sensorsfrom which the data is received are mounted in fixed positions at ornear the intersection and have at least partially non-overlapping fieldsof view. One of the sensors includes radar and the converting of thedata includes determining locations of ground transportation entitiesfrom a known location of the radar and distances from the radar to theground transportation entities.

One of the sensors includes a camera and the converting of the dataincludes determining locations of ground transportation entities from aknown location, direction of view, and tilt of the camera and thelocations of the ground transportation entities within an image frame ofthe camera.

In general, in an aspect, equipment is located at a level crossing of atransportation network that includes an intersection of a road, apedestrian crossing, and a rail line. The equipment includes inputs toreceive data from sensors oriented to monitor road vehicles andpedestrians at or near the level crossing and to receive phase andtiming data for signals on the road and on the rail line. A wirelesscommunication device is included to send to a device of one of theground transportation entities, pedestrians, or rail vehicles on therail line, a warning about a dangerous situation at or near the levelcrossing. There is storage for instructions executable by the processorto store a machine learning model that can predict behavior of groundtransportation entities at or near the level crossing at a current time.The machine learning model is based on training data about previousmotion and related behavior of road vehicles and pedestrians at or nearthe intersection. Current motion data received from the sensors aboutroad vehicles and pedestrians at or near the level crossing is appliedto the machine learning model to predict imminent behaviors of the roadvehicles and pedestrians. An imminent dangerous situation for a railvehicle on the rail line at or near the intersection is inferred fromthe predicted imminent behaviors. The wireless communication device iscaused to send the warning about the dangerous situation to a device ofat least one of the road vehicles, pedestrians, and rail vehicle.

Implementations may include one or a combination of two or more of thefollowing features. The warning is sent to an on-board equipment of therail vehicle. The rail line is on a segregated right of way. The railline is not on a segregated right of way. The equipment includes aroadside equipment. The warning is sent by broadcasting the warning forreceipt by any of the ground transportation entities, pedestrians, orrail vehicles at or near the level crossing. The imminent dangeroussituation includes a collision or a near miss.

In general, in an aspect, data is received from infrastructure sensorsrepresenting positions and motions of road vehicles being driven orpedestrians walking in a ground transportation network. Data is receivedin virtual basic safety messages and virtual personal safety messagesabout states of the road vehicles and pedestrians. The received data isapplied to a machine learning model trained to identify dangerousdriving or walking behavior of one of the road vehicles or pedestrians.The dangerous driving or walking behavior is reported to authorities.

Implementations may include one or a combination of two or more of thefollowing features. The road vehicles are identified based on platenumber recognition. The pedestrians are identified based on biometricrecognition. The road vehicles or pedestrians are identified based onsocial networking.

These and other aspects, features, and implementations can be expressedas methods, apparatus, systems, components, program products, methods ofdoing business, means or steps for performing a function, and in otherways.

These and other aspects, features, and implementations will becomeapparent from the following descriptions, including the claims.

DESCRIPTION

FIGS. 1, 2, 3, and 15 are block diagrams.

FIGS. 4, 5, 8 through 11, 13, 14, 17, and 18 are schematic views of roadnetworks from above.

FIGS. 6 and 7 are annotated perspective views of intersections.

FIGS. 12 and 16 are schematic side and perspective views of roadnetworks.

With advancements in sensor technologies and computers, it has becomefeasible to predict (and to provide early warning of) dangeroussituations and in that way to prevent collisions and near misses ofground transportation entities (that is, to enable collision avoidance)in the conduct of ground transportation.

We use the term “ground transportation” broadly to include, for example,any mode or medium of moving from place to place that entails contactwith the land or water on the surface of the earth, such as walking orrunning (or engaging in other pedestrian activities), non-motorizedvehicles, motorized vehicles (autonomous, semi-autonomous, andnon-autonomous), and rail vehicles.

We use the term “ground transportation entity” (or sometimes simply“entity”) broadly to include, for example, a person or a discretemotorized or non-motorized vehicle engaged in a mode of groundtransportation, such as a pedestrian, bicycle rider, boat, car, truck,tram, streetcar, or train, among others. Sometimes we use the terms“vehicle” or “road user” as shorthand references to a groundtransportation entity.

We use the term “dangerous situation” broadly to include, for example,any event, occurrence, sequence, context, or other situation that maylead to imminent property damage or personal injury or death and thatmay be reducible or avoidable. We sometimes use the term “hazard”interchangeably with “dangerous situation.” We sometimes use the word“violation” or “violate” with respect to behavior of an entity that has,may, or will lead to a dangerous situation.

In some implementations of the technology that we discuss here a groundtransportation network is being used by a mix of ground transportationentities that do not have or are not using transportation connectivityand ground transportation entities that do have and are usingtransportation connectivity.

We use the term “connectivity” broadly to include, for example, anycapability a ground transportation entity to (a) be aware of and act onknowledge of its surroundings, other ground transportation entities inits vicinity, and traffic situations relevant to it, (b) broadcast orotherwise transmit data about its state, or (c) both (a) and (b). Thedata transmitted can include its location, heading, speed, or internalstates of its components relevant to a traffic situation. In some cases,the awareness of the ground transportation entity is based on wirelesslyreceived data about other ground transportation entities or trafficsituations relevant to the operation of the ground transportationentity. The received data can originate from the other groundtransportation entities or from infrastructure devices, or both.Typically connectivity involves sending or receiving data in real timeor essentially real time or in time for one or more of the groundtransportation entities to act on the data in a traffic situation.

We use the term “traffic situation” broadly to include any circumstancein which two or more ground transportation entities are operating in thevicinity of one another and in which the operation or status of each ofthe entities can affect or be relevant to the operation or status of theothers.

We sometimes refer to a ground transportation entity that does not haveor is not using connectivity or aspects of connectivity as a“non-connected ground transportation entity” or simply a “non-connectedentity.” We sometimes refer to a ground transportation entity that hasand is using connectivity or aspects of connectivity as a “connectedground transportation entity” or simply a “connected entity.”

We sometimes use the term “cooperative entity” to refer to a groundtransportation entity that broadcasts data to its surroundings includinglocation, heading, speed, or states of on board safety systems (suchbrakes, lights, and wipers), for example.

We sometimes use the term “non-cooperative entity” to refer to a groundtransportation entity that does not broadcast to its surroundings one ormore types of data, such as its location, speed, heading, or state.

We sometimes use the term “vicinity” of a ground transportation entitybroadly to include, for example, an area in which a broadcast by theentity can be received by other ground transportation entities orinfrastructure devices. In some cases, the vicinity varies with locationof the entity and the number and characteristics of obstacles around theentity. An entity traveling on an open road in a desert will have a verywide vicinity since there are no obstacles to prevent a broadcast signalfrom the entity from reaching long distances. Conversely, the vicinityin an urban canyon will be diminished by the buildings around theentity. Additionally, there may be sources of electromagnetic noise thatdegrade the quality of the broadcase signal and therefore the distanceof reception (the vicinity).

As shown in FIG. 14, the vicinity of an entity 7001 traveling along aroad 7005 can be represented by concentric circles with the outermostcircle 7002 representing the outermost extent of the vicinity. Any otherentity that lies within the circle 7002 is in the vicinity of entity7001. Any other entity that lies outside the circle 7002 is outside thevicinity of, and unable to receive a broadcast by, the entity 7001. Theentity 7001 would be invisible to all entities and infrastructuredevices outside its vicinity.

Typically, cooperative entities are continuously broadcasting theirstate data. Connected entities in the vicinity of a broadcasting entityare able to receive these broadcasts and can process and act on thereceived data. If, for example, a vulnerable road user has a wearabledevice that can receive broadcasts from an entity, say an approachingtruck, the wearable device can process the received data and let thevulnerable user know when it is safe to cross the road. This operationoccurs without regard to the locations of the cooperative entity or thevulnerable user relative to a “smart” intersection as long as the user'sdevice can receive the broadcast, i.e., is within the vicinity of thecooperative entity.

We use the term “vulnerable road users” or “vulnerable road users”broadly to include, for example, any user of roadways or other featuresof the road network who is not using a motorized vehicle. vulnerableroad users are generally unprotected against injury or death or propertydamage if they collide with a motorized vehicle. In some examples,vulnerable road users could be people walking, running, cycling orperforming any type of activity that puts them at risk of directphysical contact by vehicles or other ground transportation entities incase of a collisions.

In some implementations, the collision avoidance technologies andsystems described in this document (which we sometimes refer to simplyas the “system”) use sensors mounted on infrastructure fixtures tomonitor, track, detect, and predict motion (such as speed, heading, andposition), behavior (e.g., high speed), and intent (e.g., will violatethe stop sign) of ground transportation entities and drivers andoperators of them. The information provided by the sensors (“sensordata”) enables the system to predict dangerous situations and provideearly warning to the entities to increase the chances of collisionavoidance.

We use the term “collision avoidance” broadly to include, for example,any circumstance in which a collision or a near miss between two or moreground transportation entities or between a ground transportation entityand another object in the environment that may result from a dangeroussituation, is prevented or in which chances of such an interaction arereduced.

We use the term “early warning” broadly to include, for example, anynotice, alert, instruction, command, broadcast, transmission, or othersending or receiving of information that identifies, suggests, or is inany way indicative of a dangerous situation and that is useful forcollision avoidance.

Road intersections are prime locations where dangerous situations canhappen. The technology that we describe here can equip intersectionswith infrastructure devices including sensors, computing hardware andintelligence to enable simultaneous monitoring, detection, andprediction of dangerous situations. The data from these sensors isnormalized to a single frame of reference and then is processed.Artificial intelligence models of traffic flow along differentapproaches to the intersection are constructed. These models help, forexample, entities that are more likely to violate traffic rules. Themodels are set up to detect the dangerous situations before the actualviolations and therefore can be considered as predictions. Based on aprediction of a dangerous situation, an alert is sent from theinfrastructure devices at the intersection to all connected entities inthe vicinity of the intersection. Every entity that receives an alert,processes the data in the alert and performs alert filtering. Alertfiltering is a process of discarding or disregarding alerts that are notbeneficial to the entity. If an alert is considered beneficial (i.e., isnot disregarded as a result of the filtering), such as an alert of animpending collision, the entity either automatically reacts to the alert(such as by applying brakes), or a notification is presented to thedriver or both.

The system can be used on, but is not limited to, roadways, waterways,and railways. We sometimes refer to these and other similartransportation contexts as “ground transportation networks.”

Although we often discuss the system in the context of intersections, itcan also be applied to other contexts.

We use the term “intersection” broadly to include, for example, anyreal-world arrangement of roads, rails, water bodies, or other travelpaths for which two or more ground transportation entities travelingalong paths of a ground transportation network could at some time andlocation occupy the same position producing a collision.

The ground transportation entities using a ground transportation networkmove with a variety of speeds and may reach a given intersection atdifferent speeds and times of the day. If the speed and distance of anentity from the intersection is known, dividing the distance by thespeed (both expressed in the same unit system) will give the time ofarrival at the intersection. However, since the speed of will changedue, for example, to traffic conditions, speed limits on the route,traffic signals, and other factors, the expected time of arrival at theintersection changes continuously. This dynamic change in expected timeof arrival makes it impossible to predict the actual time of arrivalwith 100% confidence.

To account for the factors affecting the motion of an entity requiresapplying a large number of relationships between the speed of the entityand the various affecting factors. The absolute values of the state ofmotion of an entity can be observed by a sensor tracking that entityeither from the entity or from an external location. The data capturedby these sensors can be used to model the patterns of motion, behaviors,and intentions of the entities. Machine learning can be used to generatecomplex models from vast amounts of data. Patterns that cannot bemodeled using kinematics of the entities directly can be captured usingmachine learning. A trained model can predict whether an entity is goingto move or stop at a particular point by using that entity's trackingdata from the sensors tracking them.

In other words, in addition to detecting information about groundtransportation entities directly from the sensor data, the system usesartificial intelligence and machine learning to process vast amounts ofsensor data to learn the patterns of motion, behaviors, and intentionsof ground transportation entities, for example, at intersections ofground transportation networks, on approaches to such intersections, andat crosswalks of ground transportation networks. Based on the direct useof current sensor data and on the results of applying the artificialintelligence and machine learning to the current sensor data, the systemproduces early warnings such as alerts of dangerous situations andtherefore aids collision avoidance. With respect to early warnings inthe form of instructions or commands, the command or instruction couldbe directed to a specific autonomous or human-driven entity to controlthe vehicle directly. For example, the instruction or command could slowdown or stop an entity being driven by a malevolent person who has beendetermined to be about to run a red light for the purpose of trying tohurt people.

The system can be tailored to make predictions for that particularintersection and to send alerts to the entities in the vicinity of thedevice broadcasting the alerts. For this purpose, the system will usesensors to derive data about the dangerous entity and pass the currentreadings from the sensors through the trained model. The output of themodel then can predict a dangerous situation and broadcast acorresponding alert. The alert, received by connected entities in thevicinity, contains information about the dangerous entity so that thereceiving entity can analyze that information to assess the threat posedto it by the dangerous entity. If there is a threat, the receivingentity can either take action itself (e.g., slowing down) or notify thedriver of the receiving entity using a human machine interface based onvisual, audio, haptic, or any kind of sensory stimulation. An autonomousentity may take action itself to avoid a dangerous situation.

The alert can also be sent directly through the cellular or othernetwork to a mobile phone or other device equipped to receive alerts andpossessed by a pedestrian. The system identifies potential dangerousentities at the intersection and broadcasts (or directly sends) alertsto a pedestrian's personal device having a communication unit. The alertmay, for example, prevent a pedestrian from entering a crosswalk andthus avoid a potential accident.

The system can also track pedestrians and broadcast information relatedto their state (position, speed, and other parameters) to the otherentities so that the other entities can take action to avoid dangeroussituations.

As shown in FIG. 1, the system includes at least the following types ofcomponents:

1. Roadside Equipment (RSE) 10 that includes or makes use of sensors 12to monitor, track, detect, and predict motion (such as speed, heading,and position), behavior (e.g., high speed), and intent (e.g., willviolate the stop sign) of ground transportation entities 14. The RSEalso includes or can make use of a data processing unit 11 and datastorage 18. The ground transportation entities exhibit a wide range ofbehavior which depends on the infrastructure of the groundtransportation network as well as the states of the entities themselves,the states of the drivers, and the states of other ground transportationentities. To capture the behaviors of the entities the RSE collectsinformation from the sensors, other RSEs, OBEs, OPEs, local or centralservers, and other data processing units. The RSE also saves the datareceived by it as well as may save the processed data at some or all thesteps in the pipeline.

The RSE may save the data on a local storage device or a remote storage.The collected data is processed in real time using predefined logic orlogic based on the data collected dynamically which means that the RSEcan update its own logic automatically. The data can be processed over asingle processing unit or a cluster of processing units to get resultsfaster. The data can be processed on a local or remote processing unitor a local or remote cluster of processing units. The RSE can use asimple logic or a sophisticated model trained on the collected data. Themodel can be trained locally or remotely.

The RSE may preprocess data before using the trained model to filteroutliers. The outliers can be present due to noise in the sensor,reflections or due to some other artifact. The resulting outliers canlead to false alarms which can affect the performance of the whole RSE.The filtration methods can be based on the data collected by the RSE,OBEs, OPEs, or online resources. The RSE may interface with othercontrollers such as traffic light controllers at the intersection orother location to extract information for use in the data processingpipeline.

The RSE also includes or can make use of communication equipment 20 tocommunicate by wire or wireless with other RSEs, and with OBEs, OPEs,local or central servers, and other data processing units. The RSE canuse any available standard for communication with other equipment. TheRSE may use wired or wireless Internet connections for downloading anduploading data to other equipment, the cellular network to send andreceive messages from other cellular devices, and a dedicated radiodevice to communicate to infrastructure devices and other RSEs at theintersection or other location.

An RSE can be installed next to different kinds of intersections. Forexample, at a signalized intersection (e.g., an intersection in whichtraffic is controlled by a light), an RSE 10 is installed near thetraffic light controllers 26 either in the same enclosure or within anearby enclosure.

Data (such as traffic light phase and timing) is meant to flow 28between the traffic light controllers and the RSE. At a non-signalizedintersection, the RSE 10 is usually located to make it easy to connectit to the sensors 12 that are used to monitor the roads or otherfeatures of the ground transportation network in the vicinity of theintersection. The proximity of RSE with the intersection helps inmaintaining a low latency system which is crucial for providing maximumtime to the receiving ground units to respond to an alert.

2. Onboard Equipment (OBE) 36 mounted on or carried by or in the groundtransportation entities 14, which includes sensors 38 that determinelocation and kinematics (motion data) of the entities in addition tosafety related data about the entities. OBEs also include dataprocessing units 40, data storage 42, and communication equipment 44that can communicate wirelessly with other OBEs, OPEs, RSEs, andpossibly servers and computing units.

3. On Person Equipment (OPE) 46 which can be, but is not limited to, amobile phone, wearable device, or any other device that is capable ofbeing worn by, held by, attached to, or otherwise interfacing with aperson or animal. OPEs can include or be coupled to data processingunits 48, data storage 50, and communication equipment 52 if needed. Insome implementations, an OPE serves as a dedicated communication unitfor a non-vehicular vulnerable road user. In some cases, the OPE canalso be used for other purposes. The OPE may have a component to providevisual, audio, or haptic alerts to the vulnerable road user.

Vulnerable road user can include pedestrians, cyclists, road workers,people on wheelchairs, scooters, self-balancing devices, battery poweredpersonal transporters, animal driven carriages, guide or police animals,farm animals, herds, and pets.

Typically an OPE is in the possession of the vulnerable road user and iscapable of sending and receiving messages. An OPE can be attached to orintegrated with a mobile phone, tablet, personal transporter, bicycle,wearable device (watch, bracelet, anklet, for example), or attached to apet collar.

Messages sent by an OPE can include kinematic information associatedwith the vulnerable road user including, but not limited to, time ofday, 3D position, heading, velocity, and acceleration. Sent messages canalso carry data representing the alertness level, current behavior, andfuture intents of the vulnerable road user, e.g. that the vulnerableroad user is currently crossing the crosswalk, is listening to music, oris going to cross the crosswalk. Among other things, the message mayconvey the blob size or data size of the vulnerable road user, whetherthere are external devices with the vulnerable road user (e.g., astroller, a cart, or other device), whether the vulnerable road user hasa disability or is using any personal assistance. The message may conveythe category of worker if the vulnerable road user is a worker and mayalso describe the type of activity being done by the worker. When acluster of similar vulnerable road users (say, a group of pedestrians)have similar characteristics, a single message can be sent to avoidmultiple message broadcasts.

Typically, the messages received by an OPE are alert messages from aroadside equipment or from an entity. The OPE can act on the receivedmessages by alerting the vulnerable road user. The alert message willcarry data useful in providing a custom alert for the vulnerable roaduser. For example, the alert to the vulnerable road user may showcase atype of dangerous situation and suggest possible actions. The OPE canapply alert filtering to all received messages and present only relevantmessages to the vulnerable road user.

Alert filtering is based on the outcome of applying a learning algorithmto historical data associated with the OPE which enablescustom-tailoring the alert filtering to each vulnerable road user. TheOPE learning algorithm tracks the responses of the vulnerable road userto received alerts and tailors future alerts to attain the best responsetime and the best attention from vulnerable road user. The learningalgorithm can also be applied to data carried in sent messages.

4. Data storage servers 54 which can be but are not limited to cloudstorage, local storage, or any other storage facility that allows forstorage and retrieval of data. The data storage servers are accessibleby RSEs, computing units, and potentially by OBEs, OPEs, and dataservers, for the purpose of storing data related to early warning andcollision avoidance, for example. The data storage servers areaccessible from RSEs and potentially from OBEs, OPEs, and data servers,for the purpose of fetching stored data. The data can be raw sensordata, processed data by a processing unit or any other informationgenerated by the RSEs, OBEs and OPEs.

Sensors at an intersection, which monitor ground transportation entitiescontinuously, can generate a large amount of data every day. The volumeof this data depends on the number and types of the sensors. The data isboth processed in real time and saved for future analysis requiring datastorage units (e.g., hard disk drives, solid state drives, and othermass storage devices) locally such as at the intersection. The localstorage devices will get filled up in a period depending on theirstorage capacity, the volume of generated data, and the rate at which itis generated. To preserve the data for future use, the data is uploadedto a remote server which has a lot more capacity. The remote server mayupgrade the storage capacity on demand as needed. The remote server mayuse a data storage device similar to the local storage (e.g., a harddisk drive, a solid state drive, or other mass storage device)accessible through a network connection.

The data stored locally and on the server for future analysis mayinclude the data broadcast by the ground transportation entities andreceived by the RSE which is saved for future analysis. The stored datacan be downloaded from the servers or other remote source for processingon the RSE. For example, the machine learning model of the intersectionwhere the RSE is located may be stored at the server or in other remotestorage, and downloaded by the RSE to use for analyzing the current datareceived at the RSE from local sources.

5. Computing units 56 which are powerful computing machines located inthe cloud or locally (for example as part of an RSE) or a combination ofthose. Among other functions, the computing units process the availabledata to generate predictions, machine learning based models of motions,behaviors, and intents of the vehicles, pedestrians, or other groundtransportation entities using the transportation network. Each of thecomputing unites can have dedicated hardware to process correspondingtypes of data (e.g., a graphics processing unit for processing images).In case of heavy processing loads, the computing unit in the RSE maybecome overloaded. This may happen, for example, when additional datageneration units (e.g. sensors) are added to the system producing acomputational overload. The overload can also occur if the logic runningin the computing unit is replaced with more computationally intensivelogic. An overload may be caused by an increase in the number of groundtransportation entities being tracked. When a local computationaloverload happens, the RSE can offload some of the tasks to anothercomputing unit. The other computing unit could be nearby the RSE orremote, such as a server. Computational tasks can be prioritized andtasks which are not time critical can be completed at the othercomputing unit and the results retrieved by the local computing unit.

For example, the computing unit in the RSE can request another computingunit to run a job for analyzing saved data and training a model usingthe data. The trained model will then be downloaded by the computingunit at the RSE to store and use there.

The computing unit at the RSE can use a other small computing units toperform a computationally intensive job more efficiently and savingtime. The available computing units are used wisely to perform the mosttasks in the least time, for example, by dividing the tasks between theRSE computing units and the other available computing units. A computingunit can also be attached as an external device to an RSE to add morecomputational capability to the computing unit in the RSE. Theexternally attached computing unit can have the same or a differentarchitecture as compared to the computing unit in the RSE. Theexternally attached computing unit may communicate with the existingcomputing unit using any available communication port. The RSE computingunit can request more computational power from the external computingunit as needed.

The rest of this document will explain in detail the roles and functionsof the components above in the system, among other things.

Roadside Equipment (RSE)

As shown in FIG. 2, an RSE may include, but not be limited to, thefollowing components:

1. One or more communication units 103, 104 which enable the receptionor transmission or both of motion data and other data related to groundtransportation entities and traffic safety data, from and to nearbyvehicles or other ground transportation entities, infrastructure, andremote servers and data storage systems 130. In some cases, this type ofcommunication is known as infrastructure-to-everything (I2X), whichincludes but is not limited to infrastructure-to-vehicles (I2V),infrastructure-to-pedestrians (I2P), infrastructure-to-infrastructure(I2I), and infrastructure-to-devices (I2D), and combinations of them.The communication may be wireless or wired and comply with a widevariety of communication protocols.

2. Communication unit 103 is used for communication with groundtransportation entities and unit 104 is used for communication throughthe Internet with remote servers and data storage systems 130.

3. Local storage 106 for storing programs, intersection models, andbehavior and traffic models. It may also be used for temporary storageof data collected from the sensors 101.

4. Sensors 101 and sensor controllers 107 which allow for the monitoringof (e.g., generating of data about) moving subjects such as groundtransportation entities typically near the RSE. The sensors may include,but are not limited to, cameras, radars, lidars, ultrasonic detectors orany other hardware that can sense or infer from sensed data the distanceto, speed, heading, location, or combinations of them, among otherthings, of a ground transportation entity. Sensor fusion is performedusing aggregations or combinations of data from two or more sensors 101.

5. A location receiver (102) (such as a GPS receiver) that provideslocalization data (e.g., coordinates of the location of the RSE)) andhelps with correcting localization errors in the localization of groundtransportation entities.

6. A processing unit 105 that will acquire and use the data generatedfrom the sensors as well as incoming data from the communication units103, 104. The processing unit will process and store the data locallyand, in some implementations, transmit the data for remote storage andfurther processing. The processing unit will also generate messages andalerts that are broadcast or otherwise sent through wirelesscommunication facilities to nearby pedestrians, motor vehicles, or otherground transportation entities, and in some cases to signs or otherinfrastructure presentation devices. The processing unit will alsoperiodically report the health and status of all the RSE systems to aremote server for monitoring.

7. Expansion connector 108 that allows for control and communicationbetween the RSE and other hardware or other components such hastemperature and humidity sensors, traffic light controllers, othercomputing units as described above, and other electronics that maybecome available in the future.

Onboard Equipment (OBE)

The onboard equipment typically may be original equipment for a groundtransportation entity or added to the entity by a third-party supplier.As shown in FIG. 3, OBE may include, but is not limited to, thefollowing components:

1. A communication unit 203 that enables the sending and receiving, orboth, of data to and from nearby vehicles, pedestrians, cyclists, orother ground transportation entities, and infrastructure, andcombinations of them. The communication unit also allows for thetransmission or reception (or both) of data between the vehicle or otherground transportation entity and a local or remote server 212 formachine learning purposes and for remote monitoring of the groundtransportation entity by the server. In some cases, this type ofcommunication is known as vehicle-to-everything (V2X), which includesbut is not limited to vehicles-to-vehicles (V2V),vehicles-to-pedestrians (V2P), vehicle-to-infrastructure (V2I),vehicle-to-devices V2D), and combinations of them. The communication maybe wireless or wired and comply with a wide variety of communicationprotocols.

Communication unit 204 will allow the OBE to communicate through theInternet with remote servers for program update, data storage and dataprocessing.

2. Local storage 206 for storing programs, intersection models, andtraffic models. It may also be used for temporary storage of datacollected from the sensors 201.

3. Sensors 201 and sensor controllers 207 that may include, but are notlimited to, external cameras, lidars, radars, ultrasonic sensors or anydevice that may be used to detect nearby objects or people or otherground transportation entities. Sensors 201 may also include additionalkinematic sensors, global positioning receivers, and internal and localmicrophones and cameras.

4. A location receiver 202 (such as a GPS receiver) that provideslocalization data (e.g., coordinates of the location of the groundtransportation entity).

5. A processing unit 205 which acquires, uses, generates, and transmitsdata, including consuming data from and sending data to thecommunication unit as well as consuming data from sensors in or on theground transportation entity.

6. Expansion connectors 208 that allows for control and communicationbetween the OBE and other hardware.

7. An interface unit that can be retrofit or integrated into ahead-unit, steering wheel, or driver mobile device in one or more wayssuch as using visual, audible, or haptic feedback).

Smart OBE (SOBE)

In a world where all vehicles and other ground transportation entitiesare connected entities, each vehicle or other ground transportationentity could be a cooperative entity with the others and could reportits current location, safety status, intent, and other information tothe others. Presently, almost all vehicles are not connected entities,cannot report such information to other ground transportation entities,and are operated by people with different levels of skill, wellbeing,stress, and behavior. Without such connectivity and communication,predicting a vehicle's or ground transportation entity's next movebecomes difficult and that translates to a diminished ability toimplement collision avoidance and to provide early warnings.

A smart OBE monitors the surroundings and users or occupants of theground transportation entity. It also keeps tabs on the health andstatus of the different systems and subsystems of the entity. The SOBEmonitors the external world by listening to, for example, the radiotransmissions from emergency broadcasts, traffic and safety messagesfrom nearby RSE, and messages about safety, locations, and other motioninformation from other connected vehicles or other ground transportationentities. The SOBE also interfaces with on board sensors that can watchthe road and driving conditions such as cameras, range sensors,vibration sensors, microphones, or any other sensor that allows of suchmonitoring. A SOBE will also monitor the immediate surroundings andcreate a map of all the static and moving objects.

A SOBE can also monitor the behavior of the users or occupants of thevehicle or other ground transportation entity. The SOBE uses microphonesto monitor the quality of the conversation. It can also use othersensors such as seating sensors, cameras, hydrocarbon sensors, andsensors of volatile organic compounds and other toxic materials. It canalso use kinematic sensors to measure the reaction and behavior of thedriver and, from that, infer the quality of driving.

SOBE also receives vehicle-to-vehicle messages (e.g., basic safetymessages (BSMs)) from other ground transport entities andvehicle-to-pedestrian messages (e.g., personal safety messages (PSMs))from vulnerable road users.

The SOBE will then fuse the data from this array of sensors, sources,and messages. It will then apply the fused data to an artificialintelligence model that is not only able to predict the next action orreaction of the driver or user of the vehicle or other groundtransportation entity or vulnerable road user, but also be able topredict the intent and future trajectories and associated near-miss orcollision risks due to other vehicles, ground transportation entitiesand vulnerable road users nearby. For example, an SOBE can use the BSMsreceived from a nearby vehicle to predict that the nearby vehicle isabout to enter into a lane change maneuver that creates a risk to itsown host vehicle, and can alert the driver of an imminent risk. The riskis computed by the SOBE based on the probability of the various futurepredicted trajectories of the nearby vehicle (e.g., going straight,changing lane to the right, changing lane to the left), and theassociated risk of collision with the host vehicle for each of thosetrajectories. If the risk of collision is higher than a certainthreshold, then the warning is displayed to the driver of the hostvehicle.

Machine learning is typically required to predict intent and futuretrajectories due to the complexity of human driver behavior modeling,which is further impacted by external factors (e.g., changingenvironmental and weather conditions).

A SOBE is characterized by having powerful computational abilities to beable to process the large number of data feeds some of which providemegabytes of data per second. The quantity of data available is alsoproportional to the level of detail required from each sensor.

A SOBE will also have powerful signal processing equipment to be able topull useful information from an environment that is known to have high(signal) noise levels and low signal to noise ratios. SOBE will alsoprotect the driver from the massive number of alerts that the vehicle isreceiving by providing smart alert filtering. The alert filtering is theresult of the machine learning model which will be able to tell whichalert is important in the current location, environmental conditions,driver behavior, vehicle health and status, and kinematics.

Smart OBEs are important for collision avoidance and early warning andfor having safer transportation networks for all users and not only forthe occupants or users of vehicles that include SOBEs. SOBEs can detectand predict the movements of the different entities on the road andtherefore aid collision avoidance.

On Person Equipment (OPE)

As mentioned earlier, on person equipment (OPE) includes any device thatmay be held by, attached to, or otherwise interface directly with apedestrian, jogger, or other person who is a ground transportationentity or otherwise present on or making use of a ground transportationnetwork. Such a person may be vulnerable road user susceptible to beinghit by a vehicle, for example. OPEs may include, but not be limited to,mobile devices (for example, smart phones, tablets, digital assistants),wearables (e.g., eyewear, watches, bracelets, anklets), and implants.Existing components and features of OPEs can be used to track and reportlocation, speed, and heading. An OPE may also be used to receive andprocess data and display alerts to the user in various modes (visual,sound, haptic, for example).

Honda has developed a communication system and method for V2Papplications focused on direct communication between a vehicle and apedestrian using OPEs. In one case, the vehicle is equipped with an OBEto broadcast a message to a surrounding pedestrian's OPE. The messagecarries the vehicle's current status including vehicle parameters,speed, and heading, for example. For example, the message could be abasic safety message (BSM). If needed the OPE will present an alert tothe pedestrian, tailored to the pedestrian's level of distraction, abouta predicted dangerous situation in order to avoid a collision. Inanother case, the pedestrian's OPE broadcasts a message (such as apersonal safety message (PSM)) to a surrounding vehicle's OBE that thepedestrian might cross the vehicle's intended path. If needed, thevehicle's OBE will display an alert to the vehicle user about apredicted hazard in order to avoid a collision. See Strickland, RichardDean, et al. “Vehicle to pedestrian communication system and method.”U.S. Pat. No. 9,421,909.

The system that we describe here, uses an I2P or I2V approach usingsensors external to the vehicle and the pedestrian (mainly oninfrastructure) to track and collect data on pedestrians and othervulnerable road users. For example the sensors can track pedestrianscrossing a street and vehicles operating at or near the crossing place.The data collected will in turn be used to build predictive models ofpedestrian and vehicle driver intents and behaviors on roads usingrule-based and machine learning methods. These models will help analyzethe data collected and make predictions of pedestrian and vehicle pathsand intents. If a hazard is predicted, a message will be broadcast fromthe RSE to the OBE or the OPE or both, alerting each entity of theintended path of the other and allowing each of them to take apre-emptive action with enough time to avoid the collision.

Remote Computing (Cloud Computing and Storage)

The data collected from the sensors connected to or incorporated in theRSEs, the OBEs, and the OPEs needs to be processed so that effectivemathematical machine learning models can be generated. This processingrequires a lot of data processing power to reduce the time needed togenerate each model. The required processing power is much more thanwhat is typically available locally on the RSE. To address this, thedata can be transmitted to a remote computing facility that provides thepower needed and can scale on demand. We refer to the remote computingfacility as a “remote server” which aligns with the nomenclature used incomputing literature. In some cases, it may be possible to perform partor all of the processing at the RCEs by equipping them with high-poweredcomputing capabilities.

Rule Based Processing

Unlike artificial intelligence and machine learning techniques,rule-based processing can be applied at any time without the need fordata collection, training, and model building. Rule-based processing canbe deployed from the beginning of operation of the system, and that itwhat is typically done, until enough training data has been acquired tocreate machine learning models. After a new installation, rules aresetup to process incoming sensor data. This is not only useful toimprove road safety but also is a good test case to make sure that allthe components of the system are working as expected. Rule basedprocessing can be also added and used later as an additional layer tocapture rare cases for which machine learning might not able to makeaccurate predictions. Rule-based approaches are based on simplerelationships between collected data parameters (e.g., speed, range, andothers). Rule-based approaches could also provide a baseline for theassessment of the performance of machine learning algorithms.

In rule-based processing, a vehicle or other ground transportationentity traversing part of a ground transportation network is monitoredby sensors. If its current speed and acceleration exceed a thresholdthat would prevent it from stopping before a stop bar (line) on a road,for example, an alert is generated. A variable region is assigned toevery vehicle or other ground transportation entity. The region islabeled as a dilemma zone in which the vehicle has not been yet labeledas a violating vehicle. If the vehicle crosses the dilemma zone into thedanger zone because its speed or acceleration or both exceed predefinedthresholds, the vehicle is labeled as a violating entity and an alert isgenerated. The thresholds for speed and acceleration are based onphysics and kinematics and vary with each ground transportation entitythat approaches the intersection, for example.

Two traditional rule-based approaches are 1) static TTI(Time-To-Intersection), and 2) static RDP (Required DecelerationParameter). See Aoude, Georges S., et al. “Driver behaviorclassification at intersections and validation on large naturalisticdata set.” IEEE Transactions on Intelligent Transportation Systems 13.2(2012): 724-736.

Static TTI (Time-To-Intersection) uses the estimated time to arrive atthe intersection as the classification criteria. In its simplest form,TTI is computed as

${{T\; T\; I} = \frac{r}{v}},$

where r is distance to the crossing line at the intersection, and v isthe current speed of the vehicle or other ground transportation entity.The vehicle is classified as dangerous if TTI<TTI_(req), where TTI_(req)is the time required for the vehicle to stop safely once braking isinitiated. The TTI_(req) parameter reflects the conservativeness levelof the rule-based algorithm. The TTI is computed on the onset ofbraking, identified as when the vehicle deceleration crosses adeceleration threshold (e.g., −0.075 g). If a vehicle never crosses thisthreshold, the classification is performed at a specified last resorttime, which typically ranges from 1 s to 2 s of estimated remaining timeto arrive at the intersection.

Static RDP (Required Deceleration Parameter) calculates the requireddeceleration for the vehicle to stop safely given its current speed andposition on the road. RDP is computed as

${{R\; D\; P} = \frac{v^{2}}{2 \times r \times g}},$

where r is distance to the crossing line at the intersection, and v isthe current speed of the vehicle or other ground transportation entity.g is the gravity acceleration constant. A vehicle is classified asdangerous (that is, the vehicle has or will create a dangeroussituation) if its required deceleration is larger than the chosen RDPthreshold RDP_(warn). In practice, a vehicle is classified as dangerousif at any time, r<r_(alert), where

$r_{alert} = {\frac{v^{2}}{2 \times {RDP}_{alert}}.}$

Similar to the static TTI algorithm, the RDP_(alert) parameter reflectsthe conservativeness level of the rule-based algorithm.

We use rule-based approaches as a baseline for the assessment of theperformance of our machine learning algorithms, and in some instances,we run them in parallel to the machine learning algorithms to capturethe rare cases in which machine learning might not able to predict.

Machine Learning

Modeling driver's behaviors have been shown to be a complex task giventhe complexity of human behavior. See H. M. Mandalia and D. D. Dalvucci,\Using Support Vector Machines for Lane-Change Detection,” Human Factorsand Ergonomics Society Annual Meeting Proceedings, vol. 49, pp.1965{1969, 2005. Machine learning techniques are well suited to modelhuman behavior but need to “learn” using training data to work properly.To provide superior detection and prediction results, we use machinelearning to model traffic detected at an intersection or other featuresof a ground transportation network during a training period before thealerting process is applied to current traffic during a deploymentphase. Machine learning can be used also to model driver responses usingin-vehicle data from onboard equipment (OBE), and could also be based onin-vehicle sensors and history of driving records and preferences. Wealso use machine learning models to detect and predict vulnerable roaduser (e.g., pedestrian) trajectories, behaviors and intents. Machinelearning can be used also to model vulnerable road users responses fromon-person equipment (OPE). These models could include interactionsbetween entities, vulnerable road users, and between one or multipleentities and one or multiple vulnerable road users.

Machine learning techniques could be also used to model the behaviors ofnon-autonomous ground transport entities. By observing or communicatingor both with a non-autonomous ground transportation entity, machinelearning can be used to predict its intent and communicate with it andwith other involved entities when a near-miss or accident or otherdangerous situation is predicted.

The machine learning mechanism works in of two phases: 1) training and2) deployment.

Training Phase

After installation, the RSE starts collecting data from the sensors towhich it has access. Since AI model training requires intensecomputational capacity, it is usually performed on powerful servers thathave multiple parallel processing modules to speed up the trainingphase. For this reason, the data acquired at the location of the RSE onthe ground transportation network can be packaged and sent to a remotepowerful server shortly after the acquisition. This is done using anInternet connection. The data is then prepared either automatically orwith the help of a data scientist. The AI model is then built to captureimportant characteristics of the flow of traffic of vehicles and otherground transportation entities for that intersection or other aspects ofthe ground transportation network. Captured data features may includelocation, direction, and movement of the vehicles or other groundtransportation entities, which can then be translated to intent andbehavior. Knowing intent, we can predict actions and future behavior ofvehicles or other ground transportation entities approaching the trafficlocation using the AI model, with high accuracy. The trained AI model istested on a subset of the data that has not been included in thetraining phase. If the performance of the AI model meets expectations,the training is considered complete. This phase is repeated iterativelyusing different model parameters until a satisfactory performance of themodel is achieved.

Deployment Phase

In some implementations, the complete tested AI model is thentransferred through the Internet to the RSE at the traffic location inthe ground transportation network. The RSE is then ready to process newsensor data and perform prediction and detection of dangerous situationssuch as traffic light violations. When a dangerous situation ispredicted, the RSE will generate an appropriate alert message. Thedangerous situation can be predicted, the alert message generated, andthe alert message broadcast to and received by vehicles and other groundtransportation entities in the vicinity of the RSE before the predicteddangerous situation occurs. This allows the operators of the vehicles orother ground transportation entities ample time to react and engage incollision avoidance. The outputs of the AI models from the variousintersections at which the corresponding RSEs are located can berecorded and made available online in a dashboard that incorporates allthe data generated and displayed in an intuitive and user-friendlymanner. Such a dashboard could be used as an interface with the customerof the system (e.g., a city traffic engineer or planner). One example ofdashboard is a map with markers that indicate the locations of themonitored intersections, violation events that have occurred, statisticsand analytics based on the AI predictions and actual outcomes.

Smart RSE (SRSE) and the Connected Entity/Non-Connected Entity Bridge

As suggested earlier, there is a gap between the capabilities andactions of connected entities and non-connected entities. For example,connected entities are typically cooperative entities that continuouslyadvertise to the world their location and safety system status such asspeed, heading, brake status, and headlight status. Non-connectedentities are not able to cooperate and communicate in these ways.Therefore, even a connected entity will be unaware of a non-connectedentity that is not in the connected entity's vicinity or out of sensorrange due to interference, distance, or the lack of a good vantagepoint.

With the proper equipment and configuration, RSEs can be made capable ofdetecting all entities using the ground transportation network in theirvicinities, including non-connected entities. Specialized sensors may beused to detect different types of entities. For example, radars aresuitable for detecting moving metallic objects such as cars, buses andtrucks. Such road entities are most likely moving in a single directiontowards the intersection. Cameras are suitable of detecting vulnerableroad users who may wander around the intersection looking for a safetime to cross.

Placing sensors on components of the ground transportation network hasat least the following advantages:

-   -   Good vantage point: Infrastructure poles, beams, and support        cables usually have an elevated vantage point. The elevated        vantage points allow for a more general view of the        intersection. This is like an observation tower at an airport        where controllers have a full view of most of the important and        vulnerable users on the ground. For ground transportation        entities, by contrast, the views from the vantage point of        sensors (camera, lidar, radar, etc. . . . or others) can be        obstructed or disrupted by a truck in a neighboring lane, direct        sunlight, or other interference. The sensors at the intersection        can be chosen to be immune or less susceptible to such        interference. A radar, for example, is not affected by sunlight        and will remain effective during the evening commute. A thermal        camera will be more likely to detect a pedestrian in a bright        light situation where the view of an optical camera becomes        hindered.    -   Fixed location: Sensors situated at the intersection can be        adjusted and fixed to sense in a specific direction that can be        optimal for detecting important targets. This will help the        processing software to better detect objects. As an example, if        a camera has a fixed view, the background (non-moving objects        and structures) information in the fixed view can be easily        detected and used to improve the identification and        classification of relatively important moving entities.

Fixed sensor location also enables easier placement of every entity in aunified global view of the intersection. Since the sensor view is fixed,the measurements from the sensor can be easily mapped to a unifiedglobal location map of the intersection. Such a unified map is usefulwhen performing global analysis of traffic movements from all directionsto study the interactions and dependencies of one traffic flow onanother. An example would be in detecting a near miss (dangeroussituation) before it happens. When two entities are traveling alongintersecting paths, a global and unified view of the intersection willenable the calculation of the time of arrival of each entity to thepoint of intersection of the respective paths. If the time is within acertain limit or tolerance, a near miss may be flagged (e.g., made thesubject of an alert message) before it happens.

With the help of the sensors that are installed on components of theinfrastructure, smart RSEs (SRSEs) can bridge this gap and allowconnected entities to be aware of “dark” or non-connected entities.

FIG. 8 depicts a scenario that explains how strategically placed sensorscan help connected entities identify the speed and location ofnon-connected entities.

A connected entity 1001, is traveling along a path 1007. The entity 1001has a green light 1010. A non-connected entity 1002 is traveling along apath 1006. It has a red light 1009 but will be making a right on redalong path 1006. This will place it directly in the path of the entity1001. A dangerous situation is imminent since the entity 1001 is unawareof the entity 1002. Because the entity 1002 is a non-connected entity itis unable to broadcast (e.g., advertise) its position and heading toother entities sharing the intersection. Moreover, the entity 1001, eventhough it is connected, is unable to “see” the entity 1002 which isobscured by the building 1008. There is a risk of the entity 1001 goingstraight through the intersection and hitting the entity 1002.

If the intersection is configured as a smart intersection, a radar 1004mounted on a beam 1005 above the road at the intersection will detectthe entity 1002 and its speed and distance. This information can berelayed to the connected entity 1001 through the SRSE 1011 serving as abridge between the non-connected entity 1002 and the connected entity1001.

Artificial Intelligence and Machine Learning

Smart RSEs also rely on learning traffic patterns and entity behaviorsto better predict and prevent dangerous situations and avoid collisions.As shown in FIG. 8, the radar 1004 is always sensing and providing datafor every entity moving along approach 1012. This data is collected andtransferred to the cloud, either directly or through an RSE, forexample, for analysis and for building and training a model that closelyrepresents the traffic along approach 1012. When the model is complete,it is downloaded to the SRSE 1011. This model can then be applied toevery entity moving along approach 1012. If an entity is classified bythe model as one that is (or is going to) violate the traffic rules, awarning (alert) may be broadcast by the SRSE to all connected entitiesin the vicinity. This warning, known as intersection collision avoidancewarning, will be received by the connected entities and can be actedupon to take account of the dangerous situation and avoid a collision.With the proper traffic model, a violating entity can be detected inadvance, giving connected entities using the intersection enough time toreact and avoid a dangerous situation.

With the help of multiple sensors (some mounted high on components ofthe infrastructure of the ground transportation network), artificialintelligence models, and accurate traffic models, an SRSE can have avirtual overview of the ground transportation network and be aware ofevery entity within its field of view including non-connected entitiesin the field of view that are not “visible” to connected entities in thefield of view. The SRSE can use this data to feed the AI model andprovide alerts to connected entities on behalf of non-connectedentities. A connected entity would not otherwise know that there arenon-connected entities sharing the road.

SRSEs have high power computing available at the location of the SRSEeither within the same housing or by connection to a nearby unit orthrough the Internet to servers. An SRSE can process data receiveddirectly from sensors, or data received in broadcasts from nearby SRSEs,emergency and weather information, and other data. An SRSE is alsoequipped with high capacity storage to aid in storing and processingdata. High bandwidth connectivity is also needed to help in transferringraw data and AI models between the SRSE and even more powerful remoteservers. SRSEs enhance other traffic hazard detection techniques usingAI to achieve high accuracy and provide additional time to react andavoid a collision.

SRSEs can remain compatible with current and new standardizedcommunication protocols and, therefore, they can be seamlesslyinterfaced with equipment already deployed in the field.

SRSEs can also reduce network congestion by sending messages only whennecessary.

Global and Unified Intersection Topology

Effective traffic monitoring and control of an intersection benefitsfrom a bird's eye view of the intersection that is not hindered byobstacles, lighting, or any other interference.

As discussed above, different types of sensors can be used to detectdifferent types of entities. The information from these sensors can bedifferent, e.g., inconsistent with respect to the location or motionparameters that its data represents or the native format of the data orboth. For example, radar data typically includes speed, distance, andmaybe additional information such as the number of moving and stationaryentities that are in the field of view of the radar. Camera data, bycontrast, can represent an image of the field of view at any moment intime. Lidar data may provide the locations of points in 3D space thatcorrespond to the points of reflection of the laser beam emitted fromthe lidar at a specific time and heading. In general, each sensorprovides data in a native format that closely represents the physicalquantities they measure.

To get a unified view (representation) of the intersection, fusion ofdata from different types of sensors is useful. For purposes of fusion,the data from various sensor is translated into a common (unified)format that is independent of the sensor used. The data included in theunified format from all of the sensors will include the global location,speed, and heading of every entity using the intersection independentlyof how it was detected.

Armed with this unified global data, a smart RSE can not only detect andpredict the movement of entities, but also can determine the relativepositions and headings of different entities with respect to each other.Therefore, the SRSE can achieve improved detection and prediction ofdangerous situations.

For example, in the scenario shown in FIG. 9, a motorized entity 2001and a vulnerable road user 2002 share the same pedestrian crossing. Theentity 2001 is traveling along a road 2007 and is detected by radar2003. The vulnerable road user 2002 walking along sidewalk 2006 isdetected by a camera 2004. The vulnerable road user 2002 may decide tocross the road 2007 using a crosswalk 2005. Doing so places the roaduser 2002 in the path of entity 2001 creating a possible dangeroussituation. If the data from each of the sensors 2003 and 2004 wereconsidered independently and no other information were considered, thedangerous situation would not be identified since each of the sensorscan only detect the entities in its respective fields of view.Additionally, each of the sensors may not be able to detect objects thatthey are not designed to detect. However, when a unified view isconsidered by the SRSE, the locations and dynamics of the entity 2001and of the vulnerable road user 2002 can be placed in the same referenceframe: a geographic coordinate system such as a map projections or othercoordinate system. When considered within a common reference system, thefused data from the sensors can be used to detect and predict adangerous situation that may arise between the two entities 2001 and2002. We will discuss the translation between the sensor space and theunified space in the following paragraphs.

Radar Data to Unified Reference Translation

As shown in FIG. 10, a radar 3001 is used to monitor road entitiestraveling along a road having two lanes 3005 and 3008 with centerlines3006 and 3007 respectively. A stop bar 3003 indicates the end of lanes3005 and 3008. T 3006 can be defined by a set of markers 3003 and 3004.FIG. 10 shows only two markers but, in general, the centerline is apiecewise linear function. The global locations of markers 3003 and 3004(and the other markers, not shown) are predefined by the design of theroadway and are known to the system. The precise global location ofradar 3001 can also be determined. Distances 3009 and 3010 of markers3003 and 3004 from the radar 3001 can, therefore, be calculated. Thedistance 3011 of the entity 3002 from the radar 3001 can be measured bythe radar 3001. Using simple geometry, the system can determine thelocation of the entity 3002 using the measured distance 3011. The resultis a global location since it is derived from the global locations ofmarkers 3003, 3004 and the radar 3001. Since every roadway can beapproximated by a generalized piecewise linear function, the methodabove can be applied to any roadway that can be monitored by a radar.

FIG. 11 shows a similar scenario on a curved road. Radar 4001 monitorsentities moving along a road 4008. The markers 4003 and 4004 represent alinear segment 4009 (of the piecewise linear function) of the centerline4007. The distances 4005 and 4006 represent the normal distance betweenthe plane 4010 of the radar 4001 and the markers 4003 and 4004respectively. Distance 4007 is the measured distance of entity 4002 fromthe radar plane 4010. Following the discussion above, given the globallocations of the radar 4001 and the markers 4003 and 4004, the globallocation of the entity 4002 can be calculated using simple ratioarithmetic.

Camera Data to Unified Reference Translation

Knowing the height, global location, direction, tilt and field of viewof a camera, calculating the global location of every pixel in thecamera image become straight forward using existing 3D geometry rulesand transformations. Consequently, when an object is identified in theimage, its global location can be readily deduced by knowing the pixelsit occupies. It is beneficial to note that the type of camera isirrelevant if its specifications are known, such as sensor size, focallength, or field of view, or combinations of them.

FIG. 12 shows a side view of a camera 5001 looking at an entity 5002.The height 5008 and tilt angle 5006 of the camera 5001 can be determinedat the time of installation. The field of view 5007 can be obtained fromthe specifications of the camera 5001. The global location of the camera5001 can also be determined at the time of installation. From the knowninformation, the system can determine the global positions of the points5003 and 5004. The distance between points 5003 and 5004 is also dividedinto pixels on the image created by the camera 5001. This number ofpixels is known from the camera 5001 specifications. The pixels occupiedby the entity 5002 can be determined. The distance 5005 can therefore becalculated. The global location of entity 5002 can also be calculated.

A global unified view of any intersection can be pieced together byfusing the information from various sensors. FIG. 13 depicts a top viewof a four-way intersection. Every leg of the intersection is divided bya median 6003. The intersection in the figure is being monitored by twodifferent types of sensors, radars and cameras, and the principlesdiscussed here can be generalized to other types of sensors. In thisexample, radar monitored regions 6001 overlap camera monitored regions6002. With a unified global view, every entity that travels betweenregions will remain tracked within the unified global view. This makesdeterminations by the SRSE, for example, of relationships between themotions of different entities easily possible. Such information willallow for a truly universal birds eye view of the intersection androadways. The unified data from the sensors can then be fed intoartificial intelligence programs as described in the followingparagraphs.

FIG. 2, discussed above illustrated components of an RSE. In addition,in an SRSE, the processing unit may also include one or severalspecialized processing units that can process data in parallel. Anexample of such units are graphic processing units or GPUs. With the aidof GPUs or similar hardware, machine learning algorithms can run muchmore efficiently at the SRSE and will be able to provide results in realtime. Such a processing architecture enables real time prediction ofdangerous situations and therefore enables sending warnings early on toallow the entities enough time to react and avoid collisions. Inaddition, because an SRSE can run processes that can use the data fromdifferent sensors and different types of sensors, the SRSE can build aunified view of the intersection that would help in the analysis oftraffic flows and the detection and prediction of dangerous situations.

Use Cases

A wide variety of cases can benefit from the system and the earlywarnings that it can provide for collision avoidance. Examples areprovided here.

Case 1: Vulnerable Ground Transportation Entities

As shown in FIG. 4, a roadway that crosses a typical intersection 409may have a pedestrian crosswalk including specific crossing areas 401,402, 403, 404 that pedestrians and other vulnerable road users(vulnerable road users) may use to walk across the roadway. Sensors thatare adequate to detect such crossings or other vulnerable users areplaced at one or more vantage points that allow the monitoring of thecrosswalk and its surroundings. During a training phase, the collecteddata can be used to train an artificial intelligence model to learnabout the behavior of vulnerable road users at the intersection. Duringa deployment phase, the AI model then can use current data about avulnerable road user to predict, for example, that the vulnerable roaduser is about to cross the roadway, and to make that prediction beforethe vulnerable road user begins to cross. When behavior and intent ofpedestrians and other vulnerable road users, drivers, vehicles, andother people and ground transportation entities can be predicted inadvance, early warnings (e.g., alerts) can be sent to any or all ofthem. Early warning can enable vehicles to stop, slow down, changepaths, or combinations of them, and can enable vulnerable road users torefrain from crossing the road when a dangerous situation is predictedto be imminent.

In general, sensors are used to monitor all areas of possible movementof vulnerable road users and vehicles in the vicinity of anintersection. The types of sensors used depend on the types of subjectsbeing monitored and tracked. Some sensors are better at tracking peopleand bicycles or other non-motorized vehicles. Some sensors are better atmonitoring and tracking motorized vehicles. The solution described hereis sensor and hardware agnostic, because the type of sensor isirrelevant if it provides appropriate data at a sufficient data ratewhich can be depend on the types of subjects being monitored andtracked. For example, Doppler radar would be an appropriate sensor tomonitor and track the speed and distance of vehicles. The data rate, orsampling rate, is the rate at which the radar is able to providesuccessive new data values. The data rate must be fast enough to capturethe dynamics of the motions of the subject being monitored and tracked.The higher the sampling rate, the more details are captured and the morerobust and accurate the representation of the motion by the databecomes. If the sampling rate is too low, and the vehicle travels asignificant distance between two sample instances, it becomes difficultto model the behavior because of the missed details during the intervalsfor which data is not generated.

For a pedestrian crossing, sensors will monitor the pedestrian and othervulnerable road users (e.g., cyclists) crossing at the intersection andthe areas in the vicinity of the intersection. The data from thesesensors may be segmented as representing conditions with respectivedifferent virtual zones to help in detection and localization. The zonescan be chosen to correspond to respective critical areas where dangeroussituations may be expected, such as sidewalks, entrances of walkways,and incoming approaches 405, 406, 407, 408 of the roads to theintersection. The activity and other conditions in every zone isrecorded. Records can include, but are not limited to kinematics (e.g.,location, heading, speed, and, acceleration) and facial and bodyfeatures (e.g., eyes, posture)

The number of sensors, number of zones, and shapes of zones are specificto every intersection and to every approach to the intersection.

FIG. 5 depicts a plan view of a typical example setup showing differentzones used to monitor and track the movement and behavior of pedestriansor other vulnerable road users, and motorized and non-motorized vehiclesand other ground transportation entities.

Sensors are set up to monitor a pedestrian crosswalk across a road.Virtual zones (301, 302) may be placed on the sidewalks and along thecrosswalk. Other sensors are placed to monitor vehicles and other groundtransportation entities proceeding on the road leading to the crosswalk,and virtual zones (303, 304) are strategically placed to aid indetecting incoming vehicles and other ground transportation entities,their distances from the crosswalk, and their speeds, for example.

The system (e.g., the RSE or SRSE associated with the sensors) collectsstreams of data from all sensors. When the system is first put intooperation, to help with equipment calibration and functionality, aninitial rule-based model may be deployed. In the meantime, sensor data(e.g., speed and distance from radar units, images and video fromcameras) is collected and stored locally at the RSE in preparation, insome implementations, to be transferred to a remote computer that ispowerful enough to build an AI model of the behavior of the differententities of the intersection using this collected data. In some cases,the RSE is a SRSE capable of generating the AI model itself.

The data is then prepared, and trajectories are built for every groundtransportation entity passing through the intersection. For example,trajectories can be extracted from radar data by stitching togetherpoints of different distances that belong to the same entity. Pedestriantrajectories and behavior can be, for example, extracted from camera andvideo recordings. By performing video and image processing techniques,the movement of the pedestrian can be detected in images and videos andtheir respective trajectories can be deduced.

For human behavior, an intelligent machine learning based modeltypically outperforms a simple rule based on simple physics. This isbecause human intent is difficult to capture, and large datasets areneeded to be able to detect patterns.

When the machine learning (AI) model is completed at the server, it isdownloaded to the RSE through the Internet, for example. The RSE thenapplies current data captured from the sensors to the AI model to causeit to predict intent and behavior, to determine when a dangeroussituation is imminent, and to trigger corresponding alerts that aredistributed (e.g., broadcast) to the vehicles and other groundtransportation entities and to the vulnerable road users and drivers asearly warnings in time to enable the vulnerable road users and driversto undertake collision avoidance steps.

This example setup can be combined with any other use case, such astraffic at signalized intersections or level crossings.

Case 2: Signalized Intersection

In the case of a signalized intersection (e.g., one controlled by atraffic light) the overall setup of the system is done as in case 1. Onedifference may be the types of sensors used to monitor or track vehiclespeed, heading, distance, and location. The setup for the pedestriancrossing of case 1 can also be combined with the signalized intersectionsetup for a more general solution.

The concept of operations for the signalized intersection use case is totrack road users around the intersection using external sensorscollecting data about the users or data communicated by the usersthemselves, predict their behaviors and broadcast alerts throughdifferent communication means about upcoming hazardous situations,generally due to violations of intersection traffic rules, such asviolating a red-light signal.

Data on road users can be collected using (a) entity data broadcast byeach entity itself about its current state, through a BSM or a PSM forinstance; and (b) sensors installed externally on infrastructure or onvehicles, such as doppler radars, ultrasonic sensors, vision or thermalcameras, lidars, and others. As mentioned earlier, the type of sensorselected and its position and orientation at the intersection shouldprovide the most comprehensive coverage of the intersection, or the partof it under study and that the data collected about the entitiesapproaching the intersection is the most accurate. Thus, the datacollected will allow reconstruction of the current states of road usersand creation of an accurate, timely, useful VBSM (virtual basic safetymessage) or VPSM (virtual personal safety message). The frequency atwhich data should be collected depends on the potential hazard of eachtype of road user and the criticality of a potential violation. Forinstance, motorized vehicles traveling at high speeds in theintersection usually require data updates 10 times per second to achievereal time collision avoidance; pedestrians crossing the intersection atmuch lower speeds can require data updates as low as 1 time per second.

As noted earlier, FIG. 4 depicts an example of a signalized intersectionplan view with detection virtual zones. These zones can segment everyapproach to the intersection into separate lanes 410, 411, 412, 413,405, 406, 407, 408 and may also separate each lane into areas thatcorrespond to general ranges of distance from the stop bar. The choiceof these zones may be performed empirically to match the character ofthe specific approaches and intersection in general. Segmenting theintersection allows for more accurate determinations of relativeheading, speed, acceleration, and positioning for each road user and inturn a better assessment of the potential hazard that road user presentsto other ground transportation entities.

In order to determine whether an observed traffic situation is adangerous situation, the system also needs to compare the outcome of thepredicted situation with the traffic light state and account for localtraffic rules (e.g. left-turn lanes, right-turn on red, and others).Therefore, it is necessary to collect and use the intersection's signalphase and timing (SPaT) information. SPaT data can be collected byinterfacing directly with the traffic light controller at theintersection, generally through a wired connection reading the data, orby interfacing with the traffic management system to receive therequired data, for instance through an API. It is important to collectSPaT data at a rate as close as possible to the rate at which road userdata is collected to ensure that road user state is always synchronizedwith traffic signal state. An added complexity to the requirement ofknowing SPaT information is that modern traffic control strategiesemployed to regulate traffic flow around intersections are not based onfixed timings and use algorithms that can dynamically adapt to real-timetraffic conditions. It is thus important to incorporate SPaT dataprediction algorithms to insure the highest accuracy in violationprediction. These SPaT data prediction algorithms can be developed usingrule-based methods or machine learning methods.

For each approach to the intersection, data is collected by the RSE (orSRSE) and a machine learning (AI) model is constructed to describe thebehavior of the vehicles corresponding to the collected data. Currentdata collected at the intersection is then applied to the AI model toproduce an early prediction whether a vehicle or other groundtransportation entity traveling on one of the approaches to theintersection is, for example, about to violate the traffic light. If aviolation is imminent, a message is relayed (e.g., broadcast) from theRSE to ground transportation entities in the vicinity. Vehicles(including the violating vehicle) and pedestrians or other vulnerableroad users will receive the message and have time to take appropriatepre-emptive measures to avoid a collision. The message can be deliveredto the ground transportation entities in one or more of the followingways, among others: a blinking light, sign, or radio signal.

If a vehicle or other entity approaching the intersection is equippedwith an OBE or an OPE, it will be able to receive the message broadcastfrom the RSE that a potential hazard has been predicted at theintersection. This allows the user to be warned and to take appropriatepre-emptive measures to avoid a collision. If the violating road user atthe intersection is also equipped with an OBE or an OPE, the user willalso receive the broadcast alert. Algorithms on the OBE or an OPE canthen reconcile the message with the violating behavior of the user andwarn the user adequately.

The decision to send an alert is dependent not only on the vehiclebehavior represented by the data collected by the sensors at theintersection. Although the sensors play a major role in the decision,other inputs are also considered. These inputs may include, but not belimited to, information from a nearby intersection (if a vehicle ran thelight at a nearby intersection, there is higher probability that itwould do the same at this intersection), information from othercooperative vehicles, or even the vehicle itself, if for example it isreporting that it has a malfunction.

Case 3: Non-Signalized Intersection

Non-signalized controlled intersections, such as a stop sign or yieldsign-controlled intersection, can be monitored as well. Sensors are usedto monitor the approach controlled by the traffic sign and predictionscan be made about incoming vehicles, similar to predictions aboutincoming vehicles on an approach to a signalized intersection. The rulesof the roads at non-signalized controlled intersections are typicallywell defined. The ground transportation entity on an approach controlledby a stop sign must come to a full stop. In a multi-way stopintersection, the right-of-way is determined by the order the groundtransportation entities reach the intersection.

A special case can be considered with a one-way stop. A set of sensorscan monitor the approach that does not have a stop sign as well. Such asetup can assist in stop sign gap negotiations. For a yield signcontrolled intersection, a ground transportation entity on an approachcontrolled by a yield sign must reduce its speed to give right-of-way toother ground transportation entities in the intersection.

A main challenge is that due to internal (e.g., driver distraction) orexternal (e.g., lack of visibility) factors, ground transportationentities violate the rules of the road, and put other groundtransportation entities at risk.

In the general case of stop-sign controlled intersections (i.e., eachapproach is controlled by a stop sign), the overall setup of the systemis done as in case 1. One difference may be the types of sensors used tomonitor or track vehicle speed, heading, distance, and location. Anotherdifference is the lack of traffic light controllers with the rules ofthe roads being indicated by the road signs. The setup for thepedestrian crossing of case 1 can also be combined with thenon-signalized controlled intersection setup for a more generalsolution.

FIG. 4 could also be understood to depict an example of a four-way stopintersection plan view with detection virtual zones. These zones cansegment every approach to the intersection into separate lanes 410, 411,412, 413, 405, 406, 407, 408 and may also separate each lane into areasthat correspond to general ranges of distance from the stop bar. Thechoice of these zones may be made empirically to match the character ofthe specific approaches and the intersection in general.

In a manner similar to the one described above for FIG. 4, current datacollected at the intersection is applied to the AI model to produce anearly prediction whether a vehicle or other ground transportation entitytraveling on one of the approaches to the intersection is about toviolate the stop sign. If a violation is imminent, messages can behandled similarly to the previously described case involving a trafficlight violation.

Also similarly to the previous description, the decision to send analert can be based on factors described previously and on otherinformation such as whether the vehicle ran the stop sign at a nearbyintersection, suggesting a higher probability that it would do the sameat this intersection).

FIG. 18 illustrates a use case for controlled non-signalizedintersection. It explains how the SRSE with strategically placed sensorscan warn a connected entity of an impending dangerous situation arisingfrom a non-connected entity.

A connected entity 9106, is traveling along a path 9109. The entity 9106has the right of way. A non-connected entity 9107 is traveling alongpath 9110. The entity 9107 has a yield sign 9104 and will be mergingonto the path 9109 without giving right of way to the entity 9106placing it directly in the path of the entity 9106. A dangeroussituation is imminent since the entity 9106 is unaware of the entity9107. Because the entity 9107 is a non-connected entity, it is unable toadvertise (broadcast) its position and heading to other entities sharingthe intersection. Moreover, the entity 9106 may not be able to “see” theentity 9107 which is not in its direct field of view. If the entity 9106proceeds along its path it may eventually have a collision with theentity 9107.

Because the intersection is a smart intersection, a radar 9111 mountedon a beam 9102 above the road will detect the entity 9107. It will alsodetect the entity 9107 speed and distance. This information can berelayed as an alert to the connected entity 9106 through the SRSE 9101.The SRSE 9101 has a machine learning model for entities moving along theapproach 9110. The entity 9107 will be classified by the model as apotential violator of the traffic rule, and a warning (alert) will bebroadcast to the connected entity 9106. This warning is sent in advancegiving the entity 9106 enough time to react and prevent a dangeroussituation.

Case 4: Level Crossings

Level crossings are dangerous because they may carry motorized vehicles,pedestrians, and rail vehicles. In many cases, the road leading to thelevel crossing falls in the blind spot of an operator (e.g., conductor)of a train or other rail vehicle. Since rail vehicle drivers operatemainly on line-of-sight information, this increases the possibility ofan accident if the road user violates the rail vehicle's right of wayand crosses a level crossing when it is not permitted to cross.

The operation of the level crossings use case is similar to thesignalized intersection use case, in the sense that a level crossing isa conflict point between road and rail traffic often regulated bytraffic rules and signals. Therefore, this use case also requirescollision avoidance warnings to increase safety around level crossings.Rail traffic can have a systematic segregated right of way, e.g.,high-speed rail, or no segregated right of way, e.g., light urban railor streetcars. With light rail and streetcars, the use case becomes evenmore important since these rail vehicles also operate on live roads andhave to follow the same traffic rules as road users.

FIG. 6 depicts a general use of a level crossing where a road and apedestrian crossing cross a railroad. Similar to the pedestrian crossinguse case, sensors are placed to collect data on the movement and intentof pedestrians. Other sensors are used to monitor and predict movementof road vehicles that are set to approach the crossing. Data on roadusers can also be collected from road user broadcasts (e.g., BSMs orPSMs). Data from nearby intersections, vehicles, and remote command andcontrol centers may be used in the decision to trigger an alert.

Data on SPaT for road and rail approaches will also need to be collectedin order to adequately assess the potential for a violation.

Similarly to the signalized intersection use case, the data collectedenables the creation of predictive models using rule-based and machinelearning algorithms.

In this use case, the rail vehicle is equipped with an OBE or an OPE inorder to receive collision avoidance warnings. When a violation of therail vehicle's right of way is predicted, the RSE will broadcast analert message, warning the rail vehicle driver that a road user is inits intended path and allowing the rail vehicle driver to takepre-emptive actions with enough time to avoid the collision.

If the violating road user is also equipped with an OBE or an OPE, themessage broadcast by the RSE will also be received by the violating roaduser. Algorithms on the OBE or an OPE can then reconcile the receivedmessage with the violating behavior of the user and warn the useradequately.

Virtual Connected Ground Transportation Environment (Bridging the Gap)

As discussed above, a useful application of the system is to create avirtual connected environment on behalf of non-connected groundtransportation entities. An impediment to the adoption of connectedtechnology is not only the absence of infrastructure installations, butalso the almost non-existence of connected vehicles, connectedvulnerable road users, and connected other ground transportationentities.

With respect to connected vehicles, in some regulatory regimes, suchvehicles are always sending what are called basic safety messages(BSMs). BSMs contain, among other information, the location, heading,speed, and future path of the vehicle. Other connected vehicles can tunein to these messages and use them to create a map of vehicles present intheir surroundings. Knowing where the surrounding vehicles are, avehicle, whether it is autonomous or not, will have information usefulto maintain a high level of safety. For example, an autonomous vehiclecan avoid making a maneuver if there is a connected vehicle in its path.Similarly, a driver can be alerted if there is some other vehicle in thepath that he is planning to follow such as a sudden lane change.

Until all ground transportation entities are equipped to send andreceive traffic safety messages and information, some road entities willbe “dark” or invisible to the rest of the road entities. Dark roadentities pose a risk of a dangerous situation.

Dark road entities do not advertise (e.g., broadcast) their location, sothey are invisible to connected entities that may expect all roadentities to broadcast their information (that is, to be connectedentities). Although onboard sensors can detect obstacles and other roadentities, the ranges of these sensors tend to be too short to beeffective in preventing dangerous situations and collisions. Therefore,there is a gap between the connectivity of connected vehicles and thelack of connectivity of non-connected vehicles. The technology describedbelow is aimed to bridge this gap by using intelligence on theinfrastructure that can detect all vehicles at the intersection or othercomponent of the ground transportation network and send messages onbehalf of non-connected vehicles.

The system can establish a virtual connected ground transportationenvironment, for example, at an intersection, that can bridge the gapbetween the future when most vehicles (and other ground transportationentities) are expected to be connected entities and the current timewhen most vehicles and other ground transportation entities have noconnectivity. In the virtual connected ground transportationenvironment, smart traffic lights and other infrastructure installationscan use sensors to track all vehicles and other ground transportationentities (connected, non-connected, semi-autonomous, autonomous,non-autonomous) and (in the case of vehicles) generate virtual BSMmessages (VBSM) on their behalf.

A VBSM message can be considered a subset of a BSM. It may not containall the fields required to create a BSM but can contain all thelocalization information including location, heading, speed andtrajectory. Since V2X communication is standardized and anonymized, VBSMand BSM cannot be differentiated easily and follow the same messagestructure. The main difference between the two messages is theavailability of the sources of the information populating thesemessages. A VBSM might lack data and information not easily generated byexternal sensors such as steering wheel angle, brake status, tirepressure or wiper activation.

With the proper sensors installed, an intersection with smart RSE candetect all the road entities that are travelling through theintersection. The SRSE can also transform all data from multiple sensorsinto a global unified coordinate system. This global unified system isrepresented by the geographical location, speed and heading of everyroad entity. Every road entity, whether it is connected or not, isdetected by the intersection equipment and a global unified location isgenerated on its behalf. Standard safety messages can, therefore, bebroadcast on behalf of the road entities. However, if the RSE broadcastsa safety message for all entities it detects, it may send a message onbehalf of a connected road entity. To address the conflict, the RSE canfilter the connected road entities from its list of dark entities. Thiscan be achieved because the RSE is continuously receiving safetymessages from connected vehicles, and the RSE sensors are continuouslydetecting road entities passing through the intersection. If thelocation of a detected road entity matches a location that from which asafety message is received by the RSE receiver, the road entity isassumed to be a connected and no safety message is broadcast on itsbehalf by the RSE. This is depicted in FIG. 15.

By creating the bridge between connected and non-connected vehicles,connected entities (including autonomous vehicles) can safely maneuverthrough intersections with complete awareness of all the road entitiesnearby.

This aspect of the technology is illustrated in FIG. 17. An intersection9001 has multiple road entities at a given time. Some of these entitiesare non-connected 9004, 9006 and others are connected 9005, 9007.Vulnerable road users 9004, 9007 are detected by a camera 9002.Motorized road entities 9005, 9006 are detected by radars 9003. Thelocation of each road entity is calculated. Broadcasts from connectedroad entities are also received by the RSE 9008. The locations ofentities from which messages are received are compared with thelocations at which entities are detected. If two entities match within apredetermined tolerance, the entity at that location is consideredconnected and no safety message is sent on its behalf. The rest of theroad entities that have no matching received location are considereddark. Safety messages are broadcast on their behalf.

For collision warnings and intersection violation warnings that are anintegral part of V2X protocols, every entity needs to be connected forthe system to be effective. That requirement is a hurdle in thedeployment of V2X devices and systems. Intersections equipped with smartRSE will address that concern by providing a virtual bridge betweenconnected and non-connected vehicles.

The US DOT (Department of Transportation) and NHTSA (National HighwayTraffic Safety Administration) identify a number of connected vehicleapplications that will use BSMs and help substantially decreasenon-impaired crashes and fatalities. These applications include, but arenot limited to, Forward Collision Warning (FCW), Intersection MovementAssist (IMA), Left Turn Assist (LTA), Do Not Pass Warning (DNPW), andBlind Spot/Lane Change Warning (BS/LCW). The US DOT and NHTSA definethese applications as follows.

An FCW addresses rear-end crashes and warns drivers of stopped, slowing,or slower vehicles ahead. An IMA is designed to avoid intersectioncrossing crashes and warns drivers of vehicles approaching from alateral direction at an intersection covering two major scenarios:Turn-into path into same direction or opposite direction and straightcrossing paths. An LTA addresses crashes where one involved vehicle wasmaking a left turn at the intersection and the other vehicle wastraveling straight from the opposite direction and warns drivers to thepresence of oncoming, opposite-direction traffic when attempting a leftturn. A DNPW assists drivers to avoid opposite-direction crashes thatresult from passing maneuvers and warns a driver of an oncoming,opposite-direction vehicle when attempting to pass a slower vehicle onan undivided two-lane roadway. A BS/LCW addresses crashes where avehicle made a lane changing/merging maneuver prior to the crashes andalerts drivers to the presence of vehicles approaching or in their blindspot in the adjacent lane.

V2X protocols stipulate that these applications should be achieved usingvehicle-to-vehicle (V2V) communications, where one connected remotevehicle would broadcast basic safety messages to a connected hostvehicle. The host vehicle's OBE would in turn try to reconcile theseBSMs with its own vehicle parameters, such as speed, heading andtrajectory and determine if there is a potential danger or threatpresented by the remote vehicle as described in the applications above.Also, an autonomous vehicle will benefit specifically from such anapplication, since it allows surrounding vehicles to communicate intent,which is a key piece of information not contained in the data collectedfrom its onboard sensors.

However, today's vehicles are not connected and, as mentioned earlier,it will take a significant period until the proportion of connectedvehicles is high for BSMs to work as explained above. Therefore, in anenvironment in which the proportion of connected vehicles is small, theconnected vehicles are not required to receive and analyze the largenumber of BSMs they would otherwise receive in an environment having aproportion of connected vehicles large enough to enable the applicationsdescribed above and benefit fully from V2X communication.

VBSMs can help bridge the gap between the current environment havinglargely unconnected entities and a future environment having largelyconnected entities and enable the applications described above, duringthe interim. In the technology that we describe here, a connectedvehicle receiving a VBSM will process it as a regular BSM in theapplications. Since VBSMs and BSMs follow the same message structure andVBS Ms contain substantially the same basic information as a BSM, e.g.,speed, acceleration, heading, past and predicted trajectory, the outcomeof applying the messages to a given application will be substantiallythe same.

For example, consider an intersection with non-protected left turns,where the connected host vehicle is about to attempt a left turn at amoment when and unconnected remote vehicle is traveling straight fromthe opposing direction with right of way. This is a situation wherecompletion of the maneuver depends on the host vehicle driver's judgmentof the situation. A wrong assessment of the situation may result in aconflict and a potential near-collision or collision. External sensorsinstalled on the surrounding infrastructure can detect and track theremote vehicle or even both vehicles, collect basic information such asspeed, acceleration, heading and past trajectory and transmit them tothe RSE, which can in turn build the predicted trajectory for the remotevehicle using rule-based or machine learning algorithms or both,populate the required fields for the VBSM and broadcast it on behalf ofthe unconnected remote vehicle. The host vehicle's OBE will receive theVBSM with information about the remote vehicle and process it in its LTAapplication to determine whether the driver's maneuver presents apotential danger and if the OBE should display a warning to the hostvehicle's driver to take preemptive or corrective action to avoid acollision. A similar result can also be achieved if the remote vehiclewere connected and received data from the RSE and the sensors that anopposing vehicle was attempting a left turn with a predicted collision.

VBSMs also can be used in lane change maneuvers. Such maneuvers can bedangerous if the vehicle changing lanes does not perform the necessarysteps to check the safety of the maneuver, e.g., check back and sidemirrors and the blind spot. new advanced driver assistance systems, suchas blind spot warnings using onboard ultrasound sensors for instance,have been developed to help prevent vehicles from performing dangerouslane changes. However, these systems can have shortcomings when thesensors are dirty or have an obstructed field of view. And existingsystems do not try to warn the endangered vehicle of another vehicleattempting a lane change. V2X communication helps solve this issuethrough applications such as BS/LCW using BSMs, however the vehicleattempting a lane change may be in unconnected vehicle and therefore notable to communicate its intent. VBSMs can help achieve that goal.Similar to the LTA use case, external sensors installed on thesurrounding infrastructure can detect and track an unconnected vehicleattempting a lane change maneuver, collect basic information such asspeed, acceleration, heading and past trajectory and transmit them tothe RSE. The RSE will in turn build the predicted trajectory for thevehicle changing lanes using rule-based and machine learning algorithms,populate the required fields for the VBSM, and broadcast it on thebehalf of the unconnected remote vehicle. The endangered vehicle's OBEwill then receive the VBSM with information about a vehicle about tomerge into the same lane, process it and determine whether the maneuverpresents a potential danger and if it should display a lane changewarning to the vehicle's driver. If the vehicle changing lanes is aconnected vehicle, its OBE can similarly receive VBSMs from the RSEabout a vehicle in its blind spot and determine whether the lane changemaneuver presents a potential danger to surrounding traffic and if itshould display a blind spot warning to the vehicle's driver. If bothvehicles are connected, both vehicles will be able to broadcast BSMs toeach other and enable BS/LCW applications. However, these applicationswill still benefit from applying the same rule-based or machine learningalgorithms (or both) on the BSM data as mentioned above to predict,early on, the intent of a vehicle changing lanes with OBEs decidingwhether to display a warning or not.

Autonomous Vehicles

The connectivity that is missing in non-connected road entities affectsautonomous vehicles.

Sensors on autonomous vehicles are either short range or have a narrowfield of view. They are unable to detect a vehicle, for example, comingaround a building on the corner of the street. They are also unable todetect a vehicle that may be hidden behind a delivery truck. Thesehidden vehicles, if they are non-connected entities, are invisible tothe autonomous vehicle. These situations affect the ability ofautonomous vehicle technology to achieve a level of safety required formass adoption of the technology. A smart intersection can help toalleviate this gap and aid acceptance of autonomous vehicles by thepublic. An autonomous vehicle is only as good as the sensors it has. Anintersection equipped with a smart RSE, can extend the reach of theonboard sensors around a blind corner or beyond a large truck. Such anextension will allow autonomous and other connected entities to co-existwith traditional non-connected vehicles. Such coexistence can acceleratethe adoption of autonomous vehicles and the advantages that they bring.

The virtual connected ground transportation environment includes VBSMmessages enabling the implementation of vehicle to vehicle (V2V),vehicle to pedestrian (V2P), and vehicle to devices (V2D) applicationsthat would have been otherwise difficult to implement.

The system can use machine learning to quickly and accurately generatethe fields of data required for the various safety messages, pack theminto a VBSM message structure and send the message to groundtransportation entities in the vicinity, using various media, such as,but not limited to, DSRC, WiFi, cellular, or traditional road signs.

Virtual Personal Safety Messages (VPMS)

The ground transportation environment can encompass not onlynon-connected vehicles but also non-connected people and othervulnerable road users.

In some regulatory regimes, connected vulnerable ground transportationentities would continuously send personal safety messages (PSMs). PSMscontain, among other information, the location, heading, speed, andfuture path of the vulnerable ground transportation entity. Connectedvehicles and infrastructure can receive these messages and use them tocreate a map that includes the vulnerable entities and enhances thelevel of safety on the ground transportation network.

Therefore, the virtual connected ground transportation environment canbridge the gap between the future when most vulnerable groundtransportation entities are expected to be connected and the currenttime when most vulnerable ground transportation entities have noconnectivity. In the virtual connected ground transportationenvironment, smart traffic lights and other infrastructure installationscan use sensors to track all vulnerable ground transportation entities(connected, non-connected) and generate VPSMs on their behalf.

A VPSM message can be considered a subset of a PSM. The VPSM need notcontain all fields required to create a PSM but can contain data neededfor safety assessment and prevention of dangerous situations and caninclude localization information including location, heading, speed, andtrajectory. In some cases, nonstandard PSM fields may also be includedin a VPSM, such as intent, posture, or direction of look of a driver.

The system can use machine learning to quickly and accurately generatethese fields, pack them into a VPSM message structure, and send it toground transportation entities in the vicinity using various media, suchas, but not limited to, DSRC, WiFi, cellular, or traditional road signs.

VPSM messages enable the implementation of pedestrian to vehicle (P2V),pedestrian to infrastructure (P2I), pedestrian to devices (P2D), vehicleto pedestrian (V2P), infrastructure to pedestrians (I2P), and devices topedestrians (D2P) applications that would have been otherwise difficultto implement.

FIG. 16 depicts a pedestrian 8102 crossing a crosswalk 8103. Thecrosswalk 8103 can be at an intersection or a mid-block crosswalk acrossa stretch of road between intersections. A camera 8101 is used tomonitor the sidewalk 8104. The global locations of the boundaries of thefield of view 8105 of the camera 8101 can be determined at the time ofinstallation. The field of view 8105 is covered by a predeterminednumber of pixels that is reflected by the specifications of camera 8101.A road entity 8102 can be detected within the field of view of thecamera and its global location can be calculated. The speed and headingof the road entity 8102 can also be determined from its displacement ats times. The path of the road entity 8102 can be represented bybreadcrumbs 8106 which is a train of locations that the entity 8102 hastraversed. This data can be used to build a virtual PSM message. The PSMmessage can then be broadcast to all entities near the intersection.

Traffic Enforcement at Non-Signalized Intersections and BehavioralEnforcement

Another useful application of the system is traffic enforcement atnon-signalized intersections (e.g. stop sign, yield sign) andenforcement of good driving behavior anywhere on the groundtransportation network.

As a byproduct of generating VBSMs and VPSMs, the system can track anddetect road users who do not abide by traffic laws and who are raisingthe probability of dangerous situations and collisions. The predictionof a dangerous situation can be extended to include enforcement.Dangerous situations need not end in collisions. Near misses are commonand can raise the stress level of drivers leading to a subsequentaccident. The frequency of near misses is positively correlated with thelack of enforcement.

Additionally, using VBSMs the system can detect improper drivingbehaviors such as abrupt lane changes and other forms of recklessdriving. The data collected by the sensors can be used to train andenable machine learning models to flag ground transportation entitiesengaging in dangerous driving behaviors.

Enforcement authorities usually enforce the rules of the roads forground transportation entities including vulnerable road users, but theauthorities need to be present in the vicinity of the intersection tomonitor, detect, and report violations. By tracking non-connected groundtransport entities including vulnerable road users using VBSMs andVPSMs, smart RSEs could play the role of enforcement authorities andenforce the rules of the roads at intersections. For example, anon-connected vehicle tracked by a smart RSE could be detected toviolate a stop or yield sign, could be identified, and could be reportedto authorities. Similarly, a vulnerable road user near an intersectiontracked by a smart RSE could be detected to unlawfully cross theintersection, could be identified, and could be reported to authorities.

For enforcement and other purposes, ground transportation entities maybe identified using unique identification including but not limited toplate number recognition. Vulnerable road users may be identified usingbiometric recognition including but not limited to facial, retina, andvoice wave identifications. In special cases that include civil orcriminal investigations, social media networks (e.g., Facebook,Instagram, Twitter) may be also used to support the identification of aviolating ground transportation entity or vulnerable road user. Anexample of leveraging social networks is to upload captured pictures ofthe violator on the social network and request users of the socialnetwork who recognize the violator to provide enforcement authoritieswith intelligence that will help identify the violator.

Other implementations are also within the scope of the following claims.

1. An apparatus comprising equipment located at an intersection of atransportation network, the equipment comprising inputs to receive datafrom sensors oriented to monitor ground transportation entities at ornear the intersection, the data from each of the sensors representing atleast one location or motion parameter of at least one of the groundtransportation entities the data from each of the sensors beingexpressed in a native format, the data received from at least two of thesensors being inconsistent with respect to the location or motionparameters or the native formats or both, a processor, and a storage forinstructions executable by the processor to convert the data from eachof the sensors into data having a common format independent of thenative formats of the data of the sensors, incorporate the data havingthe common format into a global unified representation of the groundtransportation entities being monitored at or near the intersection, theglobal unified representation including the location, speed, and headingof each of the ground transportation entities, determine relationshipsof locations and motions of two of the ground transportation entitiesusing the global unified representation, predict a dangerous situationinvolving the two ground transportation entities, and send a message toat least one of the two ground transportation entities alerting it tothe dangerous situation.
 2. The apparatus of claim 1 in which thesensors comprise at least two of: radar, lidar, and a camera.
 3. Theapparatus of claim 1 in which the data received from one of the sensorscomprises image data of a field of view at successive moments.
 4. Theapparatus of claim 1 in which the data received from one of the sensorscomprises points of reflection in 3D space.
 5. The apparatus of claim 1in which the data received from one of the sensors comprises distancefrom the sensor and speed.
 6. The apparatus of claim 1 in which theglobal unified representation represents locations of the groundtransportation entities in a common reference frame.
 7. The apparatus ofclaim 1 comprising at least two sensors from which the data is received,the two sensors being mounted in fixed positions at or near theintersection and having at least partially non-overlapping fields ofview.
 8. The apparatus of claim 7 in which one of the sensors comprisesradar and converting the data includes determining locations of groundtransportation entities from a known location of the radar and distancesfrom the radar to the ground transportation entities.
 9. The apparatusof claim 7 in which one of the sensors comprises a camera and convertingthe data includes determining locations of ground transportationentities from a known location, direction of view, and tilt of thecamera and the locations of the ground transportation entities within animage frame of the camera.
 10. A method comprising receiving data fromsensors oriented to monitor ground transportation entities at or near anintersection of a ground transportation network, the data from each ofthe sensors representing at least one location or motion parameter of atleast one of the ground transportation entities the data from each ofthe sensors being expressed in a native format, the data received fromat least two of the sensors being inconsistent with respect to thelocation or motion parameters or the native formats or both, convertingthe data from each of the sensors into data having a common formatindependent of the native formats of the data of the sensors,incorporating the data having the common format into a global unifiedrepresentation of the ground transportation entities being monitored ator near the intersection, the global unified representation includingthe location, speed, and heading of each of the ground transportationentities, determining relationships of locations and motions of two ofthe ground transportation entities using the global unifiedrepresentation, predicting a dangerous situation involving the twoground transportation entities, and sending a message to at least one ofthe two ground transportation entities alerting it to the dangeroussituation.
 11. The method of claim 10 in which the sensors comprise atleast two of: radar, lidar, and a camera.
 12. The method of claim 10 inwhich the data received from one of the sensors comprises image data ofa field of view at successive moments.
 13. The method of claim 10 inwhich the data received from one of the sensors comprises points ofreflection in 3D space.
 14. The method of claim 10 in which the datareceived from one of the sensors comprises distance from the sensor andspeed.
 15. The method of claim 10 in which the global unifiedrepresentation represents locations of the ground transportationentities in a common reference frame.
 16. The method of claim 10 inwhich the data is received from at least two sensors from the twosensors being mounted in fixed positions at or near the intersection andhaving at least partially non-overlapping fields of view.
 17. The methodof claim 16 in which one of the sensors comprises radar and the methodcomprises converting the data includes determining locations of groundtransportation entities from a known location of the radar and distancesfrom the radar to the ground transportation entities.
 18. The method ofclaim 16 in which one of the sensors comprises a camera and the methodcomprises converting the data includes determining locations of groundtransportation entities from a known location, direction of view, andtilt of the camera and the locations of the ground transportationentities within an image frame of the camera.